Using React

I finally was able to make a significant change to the app that I’ve been wanting to do for a while. I’ve separated out the REST api and the web views into separate repositories. I’ve wanted to do this since the beginning, so I kept it in mind when writing the app, and it turned out to be fairly simple to do. I had to copy a few view-related folders over, create a separate heroku app, and make sure that everything could connect. This went fairly smoothly and in the process I was able to organize a lot of code on the REST api project.

When  started the app, my goal was just to get something working from front to back, so I made the decision to not waste time trying to learn a new technology, and just went with standard html and jQuery on the front end. Now that I’ve successfully put that out, I want to move to a more modern technology, namely React. I am currently taking a few Udemy Courses related to React to get a good feel for it. The first is a straightforward intro to React and Redux. I’m not too sure what Redux is at this point, but hopefully I’ll have some idea soon, and I might be able to make use of it. I find the most difficult part of development not to be the languages themselves, but rather the dev ops – i.e. putting everything together to make it all work. Because of this, I’ve also started a Udemy course that teaches full stack development using Node and React. Based on the first few sections of these, I think they will be more than enough information for me to get going.

My plan is to create a completely new front end repo (quizipedia-react), which will call the quizipedia REST api, just like the separate html project is now doing. Once it is ready to go, I can deploy it to Heroku and deprecate the html project, although I’d like to keep it up to date provided it doesn’t take too much work.

So for October, my plan is to complete the switch from the standard html/jquery project to the react project for front-end. I’m hoping to get through the bulk of the two courses – or at least learn as much as I need to accomplish the first task Once this is done, I have a backlog of a few features I’d like to add to the project as a whole. This includes adding user accounts, new sources for the quizzes,  and custom quizzes that can be shared. I’m hoping to add these features throughout November. Then in December I’d like to create an admin site and also start looking at React Native for a mobile version of the application.

Advertisements

Wikipedia integration

I’ve added integration with wikipedia into the web app. So it is now possible to run a wikipedia search, get a list of possible articles based on your search term, and then choose one of the those articles to build a quiz out of.

I used an npm module called wikijs– This package allows you to run keyword searches against wikipedia, and also get article content in plain-text format, which is exactly what I needed. As a bonus, it also allows getting a random article, which is something I will likely incorporate in the near future. Another future development with the wiki integration is to be able to search various mediaWikis, such as the Simpsons Wiki, Star Wars wiki, etc. There is a fairly straightforward api that already exists for it, so it shouldn’t be too difficult.

But now I want to focus on cleaning up the code behind the scenes instead of adding new features. I’ve decided that my next big task is going to be to split the project up into two separate projects – one will be the RESTful service and another will be the web application. This was part of my original design, and I knew I had to do it eventually, and I think now is the right time to do it. The code base is getting larger, but more importantly, I have a very good idea of which processing belongs on the client and which belongs on the server. For example, I didn’t know where it made more sense to build the word blank. I started it on the server, then had to move it to the client, but then found I could move most of it back to the server. It would’ve taken a lot of time to tinker with this if I started with two separate projects. But now that the contracts are fairly well-established, having separate projects should make things a lot smoother without much hassle.

I don’t have a lot of experience with front-end design outside of jQuery, so I’m going to be exploring some of the big names to see which makes most sense (ember, backbone, react, angular…). I’m hoping to have this part done by the end of September. I will also likely be doing a related Udemy course alongside it.

Non-Quizipedia update

Apart from Quizipedia, I’ve also had a few other things on the go related to programming. A big one is that I’ve started doing courses on Udemy. The first one I tried was ‘Javascript: The Weird Parts’. This course digs into the guts of javascript and gives a great explanation for what it really is, and compares it to other popular programming languages. I’ve been working with javascript professionally for about a year and a half, but I really felt that I needed a better understanding of the fundamentals, and this really helped. I’ve found a big list of courses that I’m interested in taking, so I will be making updates about those as well.

I had wanted to start looking at some open-source projects. I found a few that might be interesting, but I still haven’t got a chance to contribute. Most of my “dev” time has been going towards quizipedia and the udemy course, leaving no time for an open-source project. Once I complete a few of these courses I will have another look.

Quizipedia Update

It’s been a while since I’ve posted, but I’ve been working hard on a few different aspects of Quizipedia, as well as some other minor tech ventures.

Other the past couple months, I’ve reworked the code quite a bit, although the main ideas and technologies are still intact. I’ve improved the look and controls significantly – making it nicer to look at and easier to use. I’ve made the game itself a lot easier. The first letter of a word is always given, as well as the number of letters in the word. Other new functionality includes the option to ask for a hint if you have enough points, or even solve the clue.

I’ve made some minor trial-based tweaks to the relevant word algorithm, but most importantly, I’ve integrated a dictionary into the relevant word finder. There are quite a few options out there for npm packages, but I was actually surprised to learn that a lot of the big name dictionary apis are not totally free. I found an open-source one, but it seems to be lacking a little (jupiter isn’t considered a word!), so it’s an area that can still use some improvement. I’d like to find one that uses a greater variety of proper nouns. I may even try to integrate a wikipedia or wiktionary api for the word-checking. I just wanted to try one out, so I picked a simple one, and now I can test it, and improve as needed.

Some of my goals for September include the following:

  • Integrate links (wiki, tv tropes, etc.) that will read and parse html to build quizzes
  • Add a way to generate a random/sample test, possibly from wiki or tv tropes
  • I’d like to start logging info on the relevant word scores. This would include details on each of the scoring areas for a word. Once I start gathering this data, I can use it to try and improve the algorithm. This might end up going somewhere like Elasticsearch, which I want to start learning more about
  • I am interested in starting up a new project in the near future, so I want to get quizipedia in a state where it is very easy to jump into and out of to make adjustments. This was a goal from the start, and I’ve followed it quite closely, but there are a few improvements to be made

 

Quizipedia: Writing tests

I’ve started writing some tests for Quizipedia, and I’m seeing some advantages that a test-driven approach may have had. In order to write proper tests, I’m being forced to refactor a little bit of the code. It’s not too bad, since I WAS keeping the fact that I’d need to write tests at the back of my head while I was writing the code. My main goal with this was to get SOMETHING out and working, which I was able to do. Now I get to do all the clean up. Now that I’ve scratched the itch to actually get something working, any projects I work on in the future will be test-driven (famous last words?)

Anyways, I started with the basic tests for some of the main library functions in the game (creating/getting/deleting a game and also guessing answers). I am using the chai and mocha framework to write my tests, and so far it satisfies the basic functionality that I’m looking for. Those tests were fairly straightforward, so I’ve moved on to testing the actual game-building. This includes things like scoring a given word and deciding if a word is relevant. Writing the tests allowed me to make an important refactor in the code:

Previously I had a single function that would determine the score of a word. This was a class function, since the score depended on other properties of the gameBuilder object, for example the distance to the PREVIOUS relevant word. I wanted to test the score of a word without this dependency, so I added a STATIC relevance check, which is now part of the entire relevance check. This will give greater flexibility when I integrate some more powerful word scoring into the game, so I’m glad I caught this early. I was also able to do a tiny bit of tweaking/improvements on some of the word scoring while writing the tests.

So the two library test suites are the Basic Game functionality (done) and the Game Building functionality (in progress). Once these are done, I am going to move on to writing tests for the DAL (simply sql calls => should be pretty straightforward) and then tests for the handlers (slightly more complicated).

July goals

I’m back from my trip out East and ready to get back into some existing and new projects. The PostGres Vision conference was a good opportunity to learn about a bunch of different companies that are coming up with new software using postgres. Many of the talks I went to gave an overview of some technology, followed by the presenter’s software which makes use of the technology. It wasn’t a very big conference – maybe about 200 people- so it wasn’t too overwhelming, and in that respect, it was a good place for my first tech conference.

So now that I’m back, I have some goals for July:

  • Quizipedia tests: I didn’t get a chance to write the tests that I wanted to last month, so that is going to be my priority for July. I want to have some unit and integration tests around the entire codebase, and moving forward with the project, I want to get in the habit of doing test-driven development
  • Quizipedia improvements: The relevant word algorithm on quizipedia is very primitive, so I want to do a bit of research on more intelligent ways of improving it. I may look at some of the language processing libraries that I came across while studying Postgres. This is primarily about research and investigation, s I likely won’t make any windfall changes to the code in this area.
  • Open-source project: I’m going to shop around to see if I can find an open-source project to do some work on. I’m not locking in any theme of the project, but I am going to be looking especially hard for something related to AI. These days, saying that seems like a vague statement, so I may start with some areas that I have a bit of experience in, such as constraint satisfaction problems, writing Prolog, and motion planning.
  • Books: I’ve been reading quite a few software development books lately, and I’d like to write some reviews/summaries for them.

Postgres – Pre-conference

I’m heading off for a week-long vacation out east tomorrow, and then after that I will be attending my first developer conference – the Postgresvision conference in Boston from June 27-28, so I won’t be doing much in the way of updates for the next two weeks. I didn’t make it all the way through the postgres docs, but I got the bulk of it, and it was a nice mix of refreshing my existing knowledge, clarifying some concepts I was fairly familiar with, and getting an overview of some concepts that were mostly new to me. I’m hoping to eventually get through the rest of the docs, maybe throughout some downtime during the conference.

After the conference I will write some posts about some of the lectures and workshops that I attend, and then I will hopefully be focusing on 3 areas for the month of July:

  • Finishing up some more tests and making minor improvements on quizipedia
  • Finishing up the postgres docs
  • Looking for an open-source project to get involved in

Postgres: Text Searches

So I’ve found what is easily the most interesting part of sql I’ve learned about so far: The full text search. Basically it allows you to do “intelligent (more-intelligent-than-brute-force-checking-using-the-LIKE-operator)” term searching against a longer document. The two main data types involved are the tsvector and the tsquery, which essentially represent a ‘normalized’ document and search, respectively. A tsquery contains the search query, which is searched against the tsvector.

The document to search must first be converted to the tsvector, which involves searching through the document, parsing everything into tokens (numbers, words, hyphenated words, etc). This is done by a parser. These tokens are then passed through a series of dictionaries, which attempt to find a definition for the tokens being passed in. This includes stripping a word down to a root word, and obtaining a match off that.

For example, a simple doc may contain the text: “A banker visited the Moon-Base”. The tokens produced by the parser may be something like “A”, “banker”, “visited”, “the”, “Moon-Base”. Then the dictionaries will likely resolve “banker” to the root “bank”, and it may split up “Moon-Base” into “moon” and “base”. It would also simply remove the stop words –  common words that are likely of no use (“A”, “the”…). The result of the dictionary is a set of lexemes, which just mean a normalized token. These are stored in the tsvector.

Then a query can be run against the vector to see if it matches. The tsquery can take in the standard operators (NOT, AND, OR). So something like ‘(moon | sun) & (visiting) & !gibberish’ might match, but ‘(star && moon)’ would not. There is also a ‘FOLLOWED BY’ operator (<->), to check if one word follows another. So the query ‘base <-> moon’ would not match, but ‘moon <-> base’ would.

This is just a very simple example and explanation of what it’s capable of. The best part is that this is very applicable to my Quizipedia application, and some of the terms I’ve learned about are even things that I’ve been implementing on a small-scale on my own. For example, I’ve created a list of “common words” that automatically get ignored by the relevant word algorithm. This is essentially what the dictionaries do when they catch stop words. So I will definitely be exploring this topic a lot more, and hopefully applying it to my current and future projects.

Postgres: Indexes

Today I got through the postgres chapter on indexes. I actually thought there would be a lot more to indexes, but it turns out I’m already fairly familiar with the important details. There are some index implementations of postgres that I was totally unfamiliar with (GiST, GIN, BRIN), but the B-tree is easily the most important implementation as far as I’m currently concerned.

There were a few enlightening lessons, such as the important of ordering your indexes properly, and the concept of partial indexes, but for the most part this ended up being a formalization and review.

Postgres: Functions, Operators, Conversion

Today I got through the functions, operators, and conversion sections of the postgres docs. There are lots of different functions in postgres and there is a lot of overloading on operators and functions, so this was just another overview of the possibilities. A thorough understanding of these concepts will require a lot more experience. So now, if someone asked me to name all of the functions that are associated with the bytea data type, I would still be fairly lost. BUT, my goal in doing this, is to get a general idea of the scope of problems that postgres is capable of solving. So if I arrive at some implementation problem where I need to make a scheduling and pricing system for a product, I know that postgres is ABLE to implement a money data type AND it can handle datetimes, including intervals and timezones. Postgres may not turn out to be the optimal solution for a given problem, but at least it’s another option now.