Go Struct Initialization with Named Literals

July 25th, 2014

Gopher Golang logo
The preferred way of initializing a struct is with named literals.

If the struct has non exported fields, and being called from other package, not using named literals will produce a compiler error like:

./main.go:65: too few values in struct initializer
In this case it’s required, but using them as a general practice can help avoid code breaking in the future if you rely on a struct that you may not control (3rd party) and may get changed in the future.

Example:


Go Memory Leaks detection

July 9th, 2014

Gopher Golang logo
Golang package pprof serves via its HTTP server runtime profiling data in the format expected by the pprof visualization tool.
The package is typically only imported for the side effect of registering its HTTP handlers. The handled paths all begin with /debug/pprof/.

To use pprof, import this package into your program:

import _ "net/http/pprof"

If the app is not running an http server, one needs to start one:

go func() {
  log.Println(http.ListenAndServe("localhost:6060", nil))
}()

Then its just point the browser to:

http://ip:port/debug/pprof/

More at: Golang package pprof.


Go SStore package

January 27th, 2014

TreeTimer logo
For our new Treetimer time management application, being developed mainly in Go language, we had the need to locally store simple definition files, with configurations used on the server application startup.

For that we built a very simple package called SStore, meaning “Simple Store”.
This way we can have code like this:

Configs get stored in a SStore struct type:

Then to use a config value for some key, we have:

Howto in the README file at SStore github repository.

(blogged from Vim)

Gist: Google BigQuery results reformat

September 25th, 2013

Github Octocat by Simon OxleyWorking with Google BigQuery (a database system for massive datasets, massive being billions of rows) and using Google API Client in Ruby, we noticed the responses come in a non named keys (columns) JSON format.
This new_row_format(field, rows) method solves that, rebuilding the array of hashes with named keys:

(blogged from Vim)

monstar.rb update

April 24th, 2013

New update on the monstar.rb ruby script to monitor and restart scripts/apps upon file ctime changes.Github Octocat by Simon Oxley

Now a -e –exec switch allows any system script/app to be launched via Process.spawn(SCRIPT)
PID processes are killed via INT SIGNAL.

More on github monstar.rb repository.

(blogged from Vim)

Codebits 2012

September 12th, 2012

A new year, a new Codebits event by the incredible Sapo Team.Codebits gif
I will be attending the event once again in November’12,
now that the participation was accepted for the event.
The venue is amazing, see by yourself.
Its about code, programming, web, internet, networking.


Manage URI Query params in Ruby

February 10th, 2012

How to programatically remove a Query parameter from an URI String in Ruby.Github Octocat by Simon Oxley
For the following URI:

http://example.com/resource?param1=1&param2=2&offset=3

If one wants to remove the “offset” parameter, the following may help.

EDIT: if a left side only param exists, it will be removed. Something like param2 in:
http://example.com/resource?param1=1&param2=&offset=3

(blogged from Vim)

PostgreSQL custom Restore

December 7th, 2011

On dumping and restoring PostgreSQL databases, there may be the case for restoring selected tables from a global db dump.
For this to happen one needs to have the right options when dumping the data.

pg_dump

$ pg_dump -Fc -U username database_name > db_dump

Now, we have a “restore customizable” compressed dump of the database.
From the postgres documentation:

  -F format
  –format=format
    Selects the format of the output. format can be one of the following:

    c
    custom
      Output a custom-format archive suitable for input into pg_restore.
      Together with the directory output format, this is the most flexible
      output format in that it allows manual selection and reordering of
      archived items during restore. This format is also compressed by default.

pg_restore

To reorder database items, it is first necessary to dump the table of contents of the archive into a detailed file. This file will allow us to make a custom restore of the items (tables, etc) that we need.

$ pg_restore -l db_dump > db.list

Open the db.list file in your favorite editor (Vim) and proceed to comment or delete the lines of the item that you do not want to be restored.
From the postgres documentation there is a good example I changed here to comply with the previous dump:

;
; Archive created at Wed Dec 7 15:26:56 2011
; dbname: database_name
; TOC Entries: 306
; Compression: -1
; Dump Version: 1.12-0
; Format: CUSTOM
; Integer: 4 bytes
; Offset: 8 bytes
; Dumped from database version: 9.1.1
; Dumped by pg_dump version: 9.1.1
;
;
; Selected TOC Entries:
;
3; 2615 2200 SCHEMA - public pasha
1861; 0 0 COMMENT - SCHEMA public pasha
1862; 0 0 ACL - public pasha
317; 1247 17715 TYPE public composite pasha
319; 1247 25899 DOMAIN public domain0 pasha

One can carefully edit the file. Semicolons start a comment, and the numbers at the start of lines refer to the internal archive ID assigned to each item. Lines in the file can be commented out, deleted, and reordered. According to the example, one could edit this section of the file:

10; 145433 TABLE map_resolutions postgres
;2; 145344 TABLE species postgres
;4; 145359 TABLE nt_header postgres
6; 145402 TABLE species_records postgres
;8; 145416 TABLE ss_old postgres

This way, table species will no be restored, neither nt_header and ss_old.
If one try this, will see all the items like index information, relations, keys, etc.

After all set, one can restore the selected items from the dumped database:

$ pg_restore -L db.list db_dump


Convert SQL Server database to PostgreSQL via CSV

December 2nd, 2011

Needed to convert a client Microsoft SQLSERVER database to our PostgreSQL database server.
Lots of tables, severall Gigabytes of data.

In Windows:

  • open the SQL Server administration console for the databae and do a DIR > tables.txt to get a file with all the table names;
  • remove all the tables you will not be needing from the file;
  • now to export all the tables data to their files with a one-liner:

    FOR /F %A IN ('type tables.txt') DO bcp "dbo.%A" out "d:\export\path\%A.csv" -S SERVERNAME\SQLSERVEREXPRESS -d "databasename" -U SERVERNAME\username -C RAW -T -c -t "\t" -r "\n"

  • Notice the column separator set to TAB \t, and row separator to newline \n;
  • Codepage set to RAW (this is an ISO8859-1);
  • This is done on a local machine so using a trusted connection;
  • This line is executed in the Windows console (terminal);
  • Mind that the tables.txt needs to have only the table names per line. I removed the “dbo.” prefix in the file, so had to include it in the parameter in “dbo.%A” parameter, because I want the CSV files not to have the dbo. prefix;
  • So in short, the tables.txt file is iterated, each line containing a table name being passed in the %A variables, so that bcp command creates the CSV file.

In Linux:

  • copy the previously exported data files folder to the linux machine;
  • verify the file encoding $ file -bi filename.csv
  • convert all the folder CSV files from ISO8859-1 charset to UTF-8 if needed: $ recode ISO8859-1..UTF-8 *.csv
  • now because I need to have the data validated and reassociated, I have Ruby scripts to perform the import, using Datamapper ORM. One can use the CSV class to proceed and then do whatever needs with the data. In this example I get the data into an Hash:
  • fx_tip = "./../exported_data/some_table.csv"
    h_tip = Hash.new
    CSV.foreach(fx_tip, {:col_sep => "\t", :headers => false}) do |f|
      h_tip.merge!(f[0] => f[1])
    end
    puts h_tip.to_yaml

You can even specify a manual Query to export the SQL Server data, using:
FOR /F %A IN ('type tables.txt') DO sqlcmd -E -S SERVERNAME\SQLSERVEREXPRESS -d database -o %A -s ";" -W -Q "SELECT TOP 100 * FROM dbo.%A WHERE someting"

If your case is simpler and you need to make a straight port, you can export the SQL statements from SQL Server and then on Postgres something like:
COPY tablename FROM '/tmp/table.data' WITH DELIMITER ',';

Used Tools:
Microsoft SQL SERVER
Microsoft Windows 7
PostgreSQL
Ubuntu Linux
Ruby Language

(blogged from Vim)

the R from K&R

October 13th, 2011

“Ritchie was best known as the creator of the C programming language and a key developer of the Unix operating system, and as co-author of the definitive book on C, The C Programming Language, commonly referred to as K&R (in reference to the authors Kernighan and Ritchie).”

Farewell Dennis Ritchie, your work changed my life.

C | Unix

(blogged from Vim)

“Stay Hungry, Stay Foolish”

October 6th, 2011

Dear Steve, its not easy to “Stay Hungry. Stay Foolish”… damn, it isn’t. It’s not even fun most of the time. But when it is, it certainly rocks and pays up connecting all the dots left behind. Rest in peace.

Steve Jobs


jQuery.data vs jQuery.fn.data performance

July 9th, 2011

jQuery.data( element,key,value )
“Store arbitrary data associated with the specified element. Returns the value that was set”

.data( key,value )
“Store arbitrary data associated with the matched elements.”

Performance difference: on this jsPerf test the last is about 84% slower.

So instead of:
$('#element').data('type', 'critical')
prefer:
jQuery.data('#element', 'type', 'critical')

(tested on Google Chrome 12.0.742 browser)


on client-side Cache, Sinatra and Conditional GET’s

July 1st, 2011

After re-reading Google Code - Page Speed - Optimize Caching section, I began implementing it on the Sinatra app part of our eClinic.Pro system. Later, after having it working, I found out that Sinatra has this amazing helper:

# File 'lib/sinatra/base.rb', line 290

def last_modified(time)
  return unless time
  time = time_for time
  response[’Last-Modified’] = time.httpdate
  # compare based on seconds since epoch
  halt 304 if Time.httpdate(env[’HTTP_IF_MODIFIED_SINCE’]).to_i >= time.to_i
rescue ArgumentError
end

Commited the code with a farewell git commit -am “Awesome handmade conditional GET to leverage client caching”, pushed it and refactored it to use the instance method last_modified(timestamp), just to commit it again.

Combine this with a before filter, like:

before do
  # set to rfc maximum because db_version will handle expiry date
  cache_control :public, :max_age => 31536000
  # get the database version time stamp
  db_version = Admin.first.created_at
  # now if db_version is more recent then last_modified will send a :halt 304 Not Modified as response.
  last_modified db_version
  # ... and if not, proceed here to the request route...
end

In short, the client (browser) issues a request for some content via a URI to the server. The server receives the request and checks the Request Headers to see if a If-Modified-Since directive is defined. If not, a response['Last-Modified'] header is set to the current time and the execution flow will proceed to the Sinatra app request route to deliver a response [no client local caching scenario]. Yet, this timestamp will be used by the client (browser) cache manager to store it locally, together with the response payload, building a client-side local cache for this resource. Now, if there is another client request, to the same URI, this time a If-Modified-Since Request Header will be sent along the request. The server will catch it and compare to an expiry date, in this case, the db_version. Since it is lower than, the server will stop the execution flow and send a 304 Not Modified status response. The “Conditional GET” technique name is because the GET request had an “interruption” based on conditions defined by HTTP Headers. Now, the client receiving this response will get the local storage cached content and deliver it as response. For the Sinatra app, the router was not even touched.

Acording to Google, using both Cache-Control: max-age and Last-Modified headers, or even Etag if necessary, instructs the browser to load previously downloaded resources from local disk rather than over the network. This is client (browser) caching.

Based on the documentation it is redundant to specify both Expires and Cache-Control: max-age, or to specify both Last-Modified and ETag. It makes note to use the Cache control: public directive to enable HTTPS caching for Firefox.

This allows the browser to efficiently update its cached resources by issuing conditional GET requests when the user explicitly reloads the page. Conditional GETs don’t return the full response unless the resource has changed at the server, and thus have lower latency than full GETs.

For this, I set the server to dispatch a 304 Not Modified response to the browser so that is gets the local cached content and presents it as the result of the request, with a:

Request Method: GET
Status Code: 304 Not Modified

This is quite important since I’m requesting readonly JSON content that takes about 3 seconds
to be fetched, and the result of this is 13 miliseconds response time. Local content. The payload is considerable, about 136KB of JSON response for one of the areas. Adding up that for one of the pages I’m loading 3 payloads of these via jQuery, the main server Merb app is now feeling quite lighter due to this recent decoupling. This means lower CPU utilization, database access, bandwith and time to respond.

Combining this with application server caching, response times get quite improved.

Before client (browser) local caching:
before Conditional GET

After client (browser) local caching:
after Conditional GET

Final Notes:
- Github SinatraRb documentation for (Object) last_modified(time);
- Google Code Optimize Caching page;
- HTTP Headers on Wikipedia;
- Google Chrome is an amazing browser to develop on, using the Developer Tools;
- still got to detect a “typo” on Google Code documentation.


jQuery and JSONP

June 28th, 2011

For the sake of application decoupling on one of the Merb web apps we’re working on, we identified the separation of blocks that shouldn’t depend on each other, with the advantage of having fewer dependencies, and lower the risk of malfunction in one part of a system when the other part is changed.

So a Sinatra web app was built on another server/port, with a simple yet effective control panel for internal management and an API to generate JSON responses.

Some requests to that Sinatra app API are being made via jQuery originating on the Merb application. This represents a problem. Even if in the same server, but with a different port. The browser security model tells us that XMLHttpRequest, frames, etc. must have the same domain in order to communicate. Makes sense for security reasons, but at the same time distributed services feel the pain.

For production we use a server side proxy so that the request comes from the client to our (origin) domain and is dispatched to the Sinatra app on a port or server/port combination. But in development we use another technique to get around the same origin browser policy.

That’s where JSONP (JSON with Padding) makes its debut. jQuery > 1.5 brought some new features like the ability to control the callback configuration. If we specify an ajax call/request with dataType: "jsonp", jQuery adds a parameter to our request, with the standard name “callback”. That parameter name will be used as a function name to process data.

Callback function name can be overridden using the jsonp parameter, something like jsonp: 'onJSONLoad'. That will result in the callback parameter name change. So, we’d get http://new.server:port/service/parameter?onJSONLoad= followed by an unique stamp that jQuery creates for the request.

One of the advantages of controlling the callback parameter is related to better browser caching of GET requests, since it is set to false as default for ‘jsonp’ dataType ajax requests.

All the above in the client. What about the server ? Mind that the request is getting a new parameter. For the Sinatra app, we had to change the final response. So, based on the fact of the appearance of a callback parameter:

callback = params.delete('callback')
and in the end of the action:
final_response(callback, response)
with response being a well formed JSON object.

For the final_response method, we have:

def final_response( callback, res)
  if callback
    content_type :js
    response = "#{callback}(#{res})"
  else
    content_type :json
    response = res
  end
  response
end

This will make the necessary adjustment to the response for the case of cross domain requests or same (origin) domain requests.

So a simple jQuery function would be like:


$("#ep_search_medics").click(function() {
  $.ajax({
    type: 'get',
    dataType: 'jsonp',
    url: 'http://localhost:4567/dcipt/rosuvast',
    success: function(data) { $("#ep_search_results_nr").html(data.count); },
    error: function() { console.log("Error"); }
  });
  return false;
});

Final Notes:
- one can read more at the excellent jQuery page for ajax requests;
- be very confident on your JSON response. One can test for JSON validation in JSONLint;
- Google Chrome is an amazing browser to develop on, using the Developer Tools.


DataMapper update_or_create

October 13th, 2010

Source code: git repository at github

DataMapper has a defined method named first_or_create, very explicit name, that takes a condition and attributes as params. Its source code defined in dm-core-1.0.2/lib/dm-core/model.rb is:

def first_or_create(conditions = {}, attributes = {})
  first(conditions) || create(conditions.merge(attributes))
end

What is does is simply find the first resource by conditions, or create a new resource with the attributes if none found. This always returns an instance of the resource found or created.

It became handy to create an update_or_create method, so I created:
module DataMapper
  module Model
    def update_or_create(conditions = {}, attributes = {}, merger = true)
      (first(conditions) && first(conditions).update(attributes)) || create(merger ? (conditions.merge(attributes)) : attributes )
    end
  end # Module Model
end # Module DataMapper

There are differences to consider, like the merger param and the returned TrueClass bool upon update (and not an instance of the resource).

I found a merger param necessary because of the Serial type of properties. In the implementation of first_or_create method if we specify a Serial property as a search condition and the search has no success, then the conditions will be merged with the attributes and the Serial +1 will not be taken in account. Thus, in this implementation setting merger to false, search conditions will not be merged. Some examples:

Article.update_or_create({:id => 10}, {:name => "Spirou meet Franquin"})
If Article with ‘id’ 10 exists then it will receive a new name
=> true
If Article with ‘id’ 10 doesn’t exist, it will be created with {:id => 10, :name => “Spirou meet Franquin”}
=> #[Article @id=10 @name="Spirou meet Franquin" @date=#[Date: 4910965/2,0,2299161] @updated_on=#[Date: 4910965/2,0,2299161]]

Article.update_or_create({:id => 123}, {:name => "Fantasio meet Franquin}, false)
If Article with ‘id’ 123 doesn’t exist, it will be created with {:name => “Fantasio meet Franquin”} and a self sequential id:
=> #[Article @id=240 @name="Fantasio meet Franquin" @date=#[Date: 4910965/2,0,2299161] @updated_on=#[Date: 4910965/2,0,2299161]]

(blogged from Vim)

EDIT: new implementation to always get a DataMapper Object:


install PostgreSQL in Mac OSX via Homebrew

March 10th, 2010

Apple OSX Snow Leopard
System Version: Mac OS X 10.6.2
Kernel Version: Darwin 10.2.0

Install notes for PosgreSQL 8.4 install using the great
Homebrew by Adam Vandenberg.

Run the author’s installation script, and that’s it. Try:
$ brew help
$ brew search postgresql
$ brew install postgresql
==> Summary
/usr/local/Cellar/postgresql/8.4.2: 2297 files, 31M, built in 3.3 minutes
$ brew info postgresql
postgresql 8.4.2

My advice ? Use Homebrew. True UNIX tool!

EDIT:
- sudo is no longer needed to install formulas system wide;
- the postgresql formula will be updated with recent postgres versions, so to check versions do: $ brew versions postgresql

(blogged from Vim)

install do_postgres on OSX Leopard

March 3rd, 2010

Install notes for Datamapper do_postgres driver ruby gem on severall operating systems:

Apple OSX Snow Leopard
System Version: Mac OS X 10.6.2
Kernel Version: Darwin 10.2.0
(No Macports; No Homebrew)

PosgreSQL 8.4

I wanted to have PosgreSQL server working on the mac so that development is easier.
There is a .dmg package from the guys at EnterpriseDB that is a single click install. Check it at their downloads page. Get it. Install it. Everything goes into:

/Library/PostgreSQL/8.4

Now to install the do_postgres gem, I had to set the $PATH to:

$ export PATH=$PATH:/Library/PostgreSQL/8.4/bin

Now, installing the gem from rubygems.org should work:

$ gem install do_postgres
or
sudo env ARCHFLAGS='-arch i386' gem install do_postgres

Remember in OSX where your gems are installed, with or without root access using:

$ gem environment

Debian 5.0 Lenny

I have PosgreSQL 8.3 on my Debian servers, from the repositories. To install the do_postgres gem, you must get the header files for libpq5 (PostgreSQL library):

$ sudo apt-get install libpq_dev
$ sudo gem install do_postgres

Ubuntu 9.10 Karmic Koala

I have PosgreSQL 8.4 on my Ubuntu servers, from the repositories. To install the do_postgres gem, you must get the header files for libpq5 (PostgreSQL library):

$ sudo apt-get install libpq_dev
$ sudo gem install do_postgres

(blogged from Vim)

Gemcutter indexing ruby gems

November 23rd, 2009

Date: Sat, 21 Nov 2009 22:15:15 -0500 (EST)
From: noreply@rubyforge.org
To: me@mydomain.com
Subject: [RubyForge] Gem index has been shut down

Hello -

You’re receiving this email because you’re a RubyForge project admin and a recent change robably affects you.
A few days ago we repointed gems.rubyforge.org to the gemcutter.org box. This means that Nick Quaranto’s excellent gemcutter app is now indexing and serving all the gems - so rather than having two gem indexes, we now have one. As a consequence of this, when you release files to RubyForge you will probably also want to do a “gem push” to get them onto gemcutter and into the main gem index.

Note that you can continue to release gems (and other files) at RubyForge; it’s just that gems won’t utomatically make it into the main gem index.

Yours,

The RubyForge support team
http://rubyforge.org/projects/support/


next week… Codebits

November 23rd, 2009

its next week, yet fun starts much sooner; conference; talks; code; lisbon; 3day => codebits

Just like last year a lot of hacks and experiments are expected.

Kudos to the SAPO team, the organisers, to build up such an impressive event.


Revision Control Software workshop

November 23rd, 2009

Last saturday the local Linux group, PortoLinux had a meeting at the engineering faculty of Porto University to discuss about usage of diferent SCM’s.

Main systems covered, centralized vs distributed:
- Subversion
- Git

It was a good tech event, spent a good time there.
Some photos here.


Tokyo products new site

August 31st, 2009

“By: Mikio Hirabayashi - 2009-08-31 13:22

BTW, the project site of Tokyo products have been moved onto my own server.
http://1978th.net/
Though it has not been completed yet, I’ll write more English information about Tokyo products.”

So, to get the latest, head there.


sdef - scripting definition files

August 24th, 2009

Scripting definition files (sdefs) are XML files that describe everything
about an application scripting interface: terminology, implementation
information, and complete documentation. Applications may incorporate
them to define their own scriptability, and scripting clients such as
AppleScript and Scripting Bridge read them to determine what operations
an application supports.

man sdef


Graph Database Neo4j

August 21st, 2009

Neo4j is a graph database. It is an embedded, disk-based, fully transactional Java persistence engine that stores data structured in graphs rather than in tables. A graph (mathematical lingo for a network) is a flexible data structure that allows a more agile and rapid style of development.”

From the site:

  • graph-oriented model for flexible data representation;
  • graph network consisting of nodes, relationships and properties;
  • highly scalable to bilions of nodes/rels/props;
  • can be sharded to scale out across multiple machines;
  • hi-speed traversals in the node space;
  • small footprint (500k jar)
  • ACID transactions;
  • robust persistence;
  • concurrency control;
  • API;
  • Neo4j.rb, JRuby API;
  • Lucene for quering and indexing.

Neo4j is released under a GNU Affero General Public License, so read carefully. There’s a price table for commercial products that includes severall support levels.