June Goal: PostgreSQL

At the end of June I am going to be heading to Boston for the postgresVision conference. It will be my first tech conference, and while I have a fairly good foundation in postgres, I’d still like to learn a bit more before going there. So for the month of June, I am going to be focusing on becoming an expert in postgres!

My main resource will be the official documentation found here, but I am also going to try and apply as much as possible to my postgres implementation in the quizipedia application (and of course at work). So I’ll be posting some summaries of what I learn throughout the month. After the conference, my tentative plan is to switch back to some of the larger scale goals in quizipedia, but this month will be a well-earned context-switch.

Quizipedia: Ready for alpha!

I made a few stylistic and programmatic changes to the application today. Mainly I added the first letter at the start of the blanks after immediately realizing that it was way too hard. Behind the scenes I also reworked some of the DAL code to be reusable.

I thought about some admin-related stuff, e.g. buying a domain name, or upgrading Heroku’s postGres add-on to a paid version. But I think I’d just like to get it out there and have people use it before spending any money on it.

Over the next month I’m going to gather some feedback on it, keep track of all the changes I’d like to make, and make a few updates once or twice a week. But for now, here is the link: https://go-quizipedia.herokuapp.com/

Here are my goals for the month:

  • Improve the correct/incorrect feedback to users
  • Add unit tests and integration tests
  • Setting up some “suggested” tests on the main page
  • Better handling of words that have punctuation
  • Improve the relevant word algorithm
    • Adding better regex
    • Lowering the score so that the same word doesn’t show up multiple times

 

Long term goals:

  • Importing a dictionary/thesaurus and working in multiple choice
  • Finding a way to read a pdf or other file formats
  • Adding in HTML parsing so the user only has to provide a link
  • Coming up with a way of giving the user hints

 

Quizipedia: Styling and functionality

After finishing all the connections from the views down to the database yesterday, I still need to add all the functionality of a standard web application. This includes some nice, non-invasive styling, a consistent look and feel, and some client-side AND server-side form validation. There is still lots to do on the app, but I think I’ll be ready to start showing it off to some select people this week. Right now I am just using the completely free heroku account, including the free postgres add-on. So I will likely be upgrading to some paid versions, and may even consider buying a domain name, just so I’m really going all the way with this.

I will not be heavily focusing on the app until July, so I am very glad I was able to get this far. I was hoping to get to a point where the main functionality is all there, and I can have a checklist of smaller, simpler tasks that are primarily focused on improved user interface and best coding practices. I believe I am just about there – I think one more evening of devoted work should do it. Then I’d like to break my todo-list for the app down into predictable hour-long tasks, and maybe do 2-3 every week.

These tasks will mainly include:

  • Learning and adding CSS animation
  • Automated Database scripts to remove stale data
  • Providing hints to users (maybe the first letter mininum)
  • Error logging
  •  Unit and integration tests

Quizipedia – Client-Side code

Today I was able to add the client-side views and connections to the quizipedia app. There are two simple views:

  • The game creation form: Enter a game name and the text to be parsed into a quiz.
  • The game itself: This is where the blanked out game and interaction takes place

As with the server-side code, I had written quite a bit of this code previously in another repo, so it was matter of getting it organized and hooked up with the rest of the new system. So it only took me one session to get all that done. And at this point, I’ve pretty much used up all my previously-written code, so all the next stuff is going to be new for me, and will likely take longer.

The client-side functionality is written in JavaScript, so this gives me a chance to see JavaScript in action on both the client-side and the server-side. This gets rid of any mental context-switching, which can occasionally cause a bit of lag.

So the main connections are made, and it is working well both locally and on production! So quite a satisfying day, but still lots of work to do. Over the next day or two, I’m going to get as much of the following done as I can:

  • Add in logical error-handling and logging
  • Style the client-side so it looks nice
  • Providing more feedback on correct/incorrect answers to the user
  • Add tests for the handlers
  • General code cleanup: comments, proper conventions, consistency, etc.
  • DB maintenance – maybe automate some scripts to clean up the db every so often

Once I finish a few of those tasks, I’m going to be re-focusing my goals for the month of June, which I will discuss in a later post.

Quizipedia Server-side code

I finished up most of the server-side code for my quizipedia web application today. It was fairly easy to write, since I had a few repos of sandbox code that had at least 75% of the main functionality, so it was just a matter of cleaning it up and putting it all together.

Here’s what I’ve accomplished in the past two sessions of working on it:

  • Set up a Heroku project using the node-related tutorial
    • Resource
    • Heroku makes it very easy to actually deploy something and allows you to have an easy testing environment, using a local testing database
  • Set up a postgres database (local and prod)
    • Resource
    • To add a PG db I just had to “provision” the Heroku PostGres add-on, which is straightforward and included in the node tutorial
    • My main issue was that I was trying to connect my local environment to the production database that Heroku sets up for you. It took me too long to realize that I was supposed to be using a LOCAL db, which I quickly set up
    • I think it will be important to keep a change log of any db updates that I do. When making changes locally, I can just wipe the db and re-run the script I’ve saved. But this isn’t going to work on prod, where the existence of the data depends on me NOT destroying the tables. I have some experience using a migrations package, which allows you to create a new ‘migration’ file anytime you need to make a change, for example adding a table column. This will be essential for keeping track of all my changes. I’m not going to add it in right now, as my focus is to get something out and working.
  • Added the main handlers. This is a RESTful application, so the primary handlers right now include:
    • POST /game: This is where a user would post some text to be formatted into a quiz. Eventually I would like to have them just send a link to a webpage with the data, do the parsing, and create the quiz that way.
    • GET /game/:gameId: This is going to fetch the game information by the id generated in the saving functionality. It will return the blanked out game. Further parsing will be done on the client-side
    • POST /guess: This is is a call to allow the user to submit a guess for one of the blanks. Their guess will be verified against the true answer. If it is correct, it will be saved accordingly.
  • Connected the handlers to the DB via a function library and a data layer.
    • The data layer is essentially an active record setup, where the models reflect the columns in the database. I have three tables, so I have 3 dal classes.
    • The library is the business logic layer, where the relevant word algorithm gets used, and the objects get converted to the appropriate formats
    • Building the game is a large job, so I created a completely separate ‘gameBuilder’ helper class which is called by the main ‘game’ library class.

 

So I’m happy with this initial progress. Again, a lot of it was just organizing some existing code in some unorganized repos into something workable and organized.

My next big step is going to be to add the client-side handling. This will mainly involve submitting a new game text, presenting the constructed game, and filling out correct answers as needed.

Quizipedia

My current hobby project is called Quizipedia. It begins with a  text article, maybe from wikipedia, another url, or maybe even a pdf. It then uses an algorithm to find ‘relevant words’ in that article. These words are then blanked out and the user is presented with a fill-in-the-blanks quiz where they must complete the missing words.

First question is how do we determine if a word is relevant. Without going into too many details, some of the factors that determine whether or not a word is relevant include:

  • Word length (longer words are more relevant)
  • Capitalization (Capitalized words are more relevant)
  • Occurrences (if it appears many times, then it is less relevant)
  • String searches (words ending in ‘ly’ are usually just adverbs, so therefore less relevant)

There are likely many other factors that may contribute to a word’s relevancy. My main goal right now is to just turn this into a simple web application so that others can try it out.

Since this is my first major solo project, I’ve decided to use some tools that I’m familiar with and interested in learning more about. It will be a standard three-tiered application structures as follows:

  • Front End: html, ejs, jquery – all using the express framework
  • Application layer: Node.js
  • Data layer: PostGres

My goal is to constantly update this blog with my progress on the app. Once I reach certain milestones in the development, I would like to post some tutorials based on what I’ve learned. Stay tuned!

 

Best Practices

Some simple, yet sometimes-forgettable practices to remember:

  • 4 attributes of good code: Maintainable, efficient, usable, dependable
  • Establish programming conventions before beginning
  • Make code portable. Don’t hard-code values
    • SPOD: Single Point of Definition
  • Be pro-active with tests. Use Test-Driven Development
  • Deployment considerations:
    • Keep only what is needed – nothing unused
    • Have a roll-back strategy
    • Use automation for repeatable processes – eliminates human error
  • Traditional Waterfall approach
    • Requirements, Design, write code, subsystem testing, system testing
    • The assumption is that this process is rigidly consecutive, e.g. Once requirements are done, they will never change. This does NOT reflect reality
  • General problems:
    • Poor code quality
    • Unmaintainable
    • Inaccurate understanding of user needs
    • Inability to deal with changing requirements
  • Specific problems:
    • Ambiguous communication
    • Insufficient testing
    • Bad requirement management
    • Undetected inconsistencies in requirements
  • Best practices
    • Develop iteratively
      • Resolve critical risk before making large investment
      • Early iterations promote early feedback
    • Manage requirements
      • Expect them to change
      • Agree with user on the WHAT, not the HOW
      • Prioritize the requirement changes
    • Component architecture
      • Permits re-use
      • Allows use of commercial software and APIs
      • Easy to break down tasks for a team
      • Improves maintainability and extensibility
    • Model software visually
      • Promotes unambiguous communication
      • Can hide/expose details in accordance with the requirements

Sources:

Wiki

Lecture

Blogs and Podcasts

This is a list of blogs and podcasts related to software engineering and technology in general that I’m trying to follow regularly:

Blogs:

  • Coding Horror: https://blog.codinghorror.com/
  • Scott Hanselman: https://www.hanselman.com/
  • JavaScript Tips: http://www.jstips.co/
  • David Walsh Blog: https://davidwalsh.name/
  • Daily WTF: http://thedailywtf.com/

 

Podcasts:

  • Hanselminutes
  • Data Skeptic
  • Partially Derivative
  • Wall Street Journal Tech News Briefing
  • Software Engineering Radio

Intro

Hello! Welcome to my programming blog. This is a record of my quest to become a professional software developer. I have a computer science/math combined degree and have been working in industry as a developer for just under two years at the time of writing.

My goal is to continue learning about all aspects of software engineering. This includes but is not limited to the following:

  • Professionalism and ethics
  • Law and regulation
  • Mathematical foundation
  • Technical knowledge
    • Programming basics and core concepts
    • Various languages
    • Databases
    • AI (personal interest)
  • Best practices