📩 about đŸ§» posts
Garry Newman
🏠 Home 📄 Posts

Projects

Facepunch Garry's Mod Rust S&box

Online

Twitter-color Created with Sketch. Twitter YouTube TikTok GitHub
Posts
How do I get into the Game Industryone month agoquestion_answer 14In The Shit2 months agoquestion_answer 7Dead Games4 months agoquestion_answer 9Azure Media Services11 months agoquestion_answer 13Common Math Library11 months agoquestion_answer 6Handshake Gameone year agoquestion_answer 3The Death Of The Webone year agoquestion_answer 13Aaron Petersone year agoquestion_answer 19Your licensing system sucksone year agoquestion_answer 11Disposable Actionone year agoquestion_answer 5The Good Guysone year agoquestion_answer 15Unity can get fuckedone year agoquestion_answer 66Facepunch Birminghamone year agoquestion_answer 10Mexicoone year agoquestion_answer 8NPM2 years agoquestion_answer 4Facepunch Bloxwich2 years agoquestion_answer 2Lapland2 years agoMailbag: Sausage Men2 years agoquestion_answer 2Holiday GameDev Books2 years agoOffice Design2 years agoAccess control in serverside Blazor2 years agothe man who fell to earth2 years agoquestion_answer 1Unity 20222 years agoquestion_answer 13D Printer2 years agoquestion_answer 1Rip Foszor3 years agoquestion_answer 2My Tesla Is Fixed3 years agoMy Tesla Broke3 years agoWhat sucks about Steam3 years agoquestion_answer 1Making A Game 20223 years agoquestion_answer 3InteropGen3 years agoFrozen Shoulder3 years agoShitty Holiday Photos3 years agoYour Portfolio Gave Me Diarrhea3 years agoquestion_answer 2Motivation3 years agoquestion_answer 1We got married3 years agoEverything We Watched In Lockdown4 years agoHouse - What I'd Have Done Different4 years agoHouse - Fuckups4 years agoReceiver Broken4 years agoMy Feet Are Fucked4 years agoWhat Unity Is Getting Wrong5 years agoquestion_answer 4Lockdown Kicks Ass5 years agoHalf-Life:Alyx is a Masterpiece5 years agoOpen a 30gb Unity Log on Windows5 years agoAlex The Gamer5 years agoBlazor Ain't Ready5 years agoJailbreaking Control45 years agoBenchmarking Rust5 years ago£600 Remote5 years agoRusty Legs5 years agoControl45 years agoAndroid: Best & Worst5 years agoRobot Lawnmower6 years agoBest VR Shagging Games6 years agoquestion_answer 2Red Dead 26 years agoUnity FPSSample6 years agoForum Downtime6 years agoFatty6 years agoBlogging & Twitter6 years agoUnity's Staging Packages7 years agoPlaytest Recording7 years agoSteamworks and IL2CPP7 years agoMonitor Upgrade7 years agoSteamworks Libraries7 years agoProcedural Generation7 years agoUnity 2018.17 years agoTimeSince7 years agoKitchen Wank7 years agoPrototyping7 years agoDisplaying an Avatar7 years agoPlayer Name Overlay7 years agoTesla Model X7 years agoNaming Stuff7 years agoFacepunch Api8 years agoAggregation8 years agoThe curse of Early Access8 years agoJenkins Library8 years agoJenkinsfile8 years agoJenkins8 years agoUnity 2017 Changelist8 years agoBinding Steamworks SDK8 years agoFacepunch.Steamworks8 years agoPolitics8 years agoUnreal Engine8 years agoI got suspended from twitter8 years agoPlasticSCM Speed Tip8 years agoBecoming A Gamedev9 years agoJessica9 years agoLiving Room Vive9 years agoI jacked off in VR, twice9 years agoquestion_answer 6GDC 20169 years agoRust's Networking9 years agoCrash Log Tags9 years agoAutomated Crash Processor9 years agoMicrotransactions9 years agoUnity Tips 210 years agoUnity Tips10 years agoGodus10 years agoSteam10 years agoPaying for Mods10 years agoLeafblower10 years agoGarry's Mod is 10 years old10 years agoquestion_answer 1Selling Out10 years agoOne Year On10 years agoI fell out of love with Lua10 years agoRust Screenshots10 years agoDepression10 years agoHow I Upload Stuff11 years ago9 reasons why I LOVE C#11 years agoSerializing Shit11 years agoUnity Viewmodels11 years agoquestion_answer 1Baby Gadgets11 years agoValve VR Demo11 years agoSteam Dev Days : Day 211 years agoSteam Dev Days Day 111 years ago2013 Goals - in review11 years agoCatchup11 years agoIs Paypal Slow11 years agoStop Programming11 years agoGrand Design11 years agoNext Gen Console11 years agoParenting11 years agoChildbirth II11 years agoChild Birth11 years agoResuable Tools11 years agoDesign by Committee11 years agoThe Columbo11 years agoThis Old House11 years agoClustered11 years agoThe Rust backend11 years agoServer Downloads11 years agoThe Story of Rust12 years agoquestion_answer 2The Conversion12 years agoLike a Rolling Stone12 years agom0r3 gmod beta12 years agoSteampipe Beta12 years agoPiss Shake12 years agoquestion_answer 1The Pipe Beta!12 years agoOculus Rift12 years agoLinux and The Pipe12 years agoOffice downstairs12 years agoUnity Wordpress Plugin12 years agoLinuxxx12 years agoThe Beach12 years agoTrees have leaves12 years agoOBS12 years agoPrivacy Thing12 years agoOffice Progress12 years agoLearning Unity12 years agoNew Project12 years agoNextbot & Coroutines12 years agoAirplay To Room12 years agoResellers12 years agoSteampipe Looming12 years agoCoroutines12 years agoRecruiters12 years agoWriting a Wiki Bot12 years agoOil your brain12 years agoTax Investigation12 years agoLua Bindings12 years agoOffice Progress12 years agoPaths12 years agoNext Bot12 years agoThe Sign12 years agoVPK Search Paths12 years agoMailbag12 years agoSteamPipe Mounting12 years agoMailbag – Questions12 years agoOffice Refurb Progress12 years agoTesting GMod Updates12 years agoCatch Up12 years agoGit outa hear12 years agoShadow Gap12 years ago2013 Goals12 years agoPC Hardware Design Sucks12 years agoBorderless Window Mode?12 years agoMore Kinect Stuff12 years agoCoD or Battlefield?12 years agoKinecting GMod12 years agoEntity Editor12 years agoFan Mail12 years agoLua Errors12 years agoOptimising GMod12 years agoTexture Filtering12 years agoToybox Torrent12 years agoToybox Disabled12 years agoquestion_answer 1Snack Chair12 years agoCoffee Shirt12 years agoCrash Dump12 years agoLuaJIT12 years agoThanks EU12 years agoNew Office12 years agoSky Shader12 years agoEgypt!12 years agoquestion_answer 1I'm going to Egypt for 2 weeks12 years agoMake maps bigger12 years agoApproved Updates12 years agoquestion_answer 1Context Screen Improvements12 years agoAuto Refresh12 years agoGMod 13 Release Date12 years agoPyro Patch12 years agoSFM vs GMod12 years agoMore Dieting13 years agoAmazon Cloud Search13 years ago13 years agoFrame Blending13 years agoDemo Movies13 years agoDOTA213 years agoWeight Loss Update13 years agoCheaters in GMod13 years agoPortable Computer13 years agoCoding Websites in C++13 years agoBigger Maps13 years agoSteam Captcha13 years agoQuitting Caffine13 years agoMaking Gamemodes Better13 years agoLottery Results13 years agoGWEN Designer13 years agoUnclamped Faceposer13 years agoThe Lottery13 years agoPoster Screenshots13 years agoTrade Off Game Dev13 years agoVersion 49 Models13 years agoImproved Shadows13 years agoquick tool: bundler13 years agoWeight Loss13 years agoWorkshopping13 years agoMore Stuff I did13 years agoCool stuff I did this week13 years agoMegaUpload13 years agoOops13 years agoSOPA13 years agoFamily Tree13 years agoTop 6 Games of 201113 years agoAnt Farm 2.013 years agoCoding Evolution13 years agoHolly Holly, Get Dolly13 years agomklink13 years agoVideo13 years agoBlog update13 years agoAss Kicker13 years agoOther Stuff13 years agoStuff done did13 years agoI done maths13 years agoPC Specs13 years agoBattlefield13 years agoBeta Keys Depleted13 years agoCoding & Beta13 years agoHow To Buy Hardware13 years agoBodygroups13 years agoProgress13 years agoNumpads and gibs13 years agoSpawnmenu update13 years agoBack in business13 years agoWe Beat Righthaven!13 years agoBack from Holiday!13 years agoHoliday Continue13 years agoHoliday Halt13 years agoVegas13 years agoHoliday13 years agoStill Coding13 years agoBeing Sued13 years agoSpawnicons13 years agoOffice Dog13 years agoThe Plan13 years agoErection Failure13 years agoquestion_answer 2The Wild13 years agoGWEN Skins / Pixel Reading13 years agoWIT: Deus Ex (3 I think)13 years agoCODBLOPS13 years agoResources13 years agoDocking13 years agoSteam Chat13 years agoSelling a Car Online13 years agoMy First PC Games13 years agoWIT: From Dust13 years agoThat GMod Update13 years agoGMod Broke Routine13 years agoCloudflare security issues13 years agoImages in Source13 years agoVisual Studio for Lua editing13 years agoMore Android bitching13 years agoRiots13 years agoHome Gym13 years agoAndroid vs iOS13 years agoDoing Stuff13 years agoBlogging lookback13 years agoFrom wordpress to tumblr13 years agoGMod Updating13 years agoBullshit13 years agoForum Owner13 years agoSarah, Part 315 years agoSarah, part 215 years agoSarah, part 115 years agoThe Perfect Woman15 years agoquestion_answer 3The Duplicator18 years agoSTALKER18 years agoSUCCESS!18 years agoReleased!18 years agoRelease Today!18 years agoGMod10 - Tomorrow!18 years agoquestion_answer 1Back In England!19 years agoNew Lua Implementation19 years agoProfessional Mod Makers19 years agoGrey Hair19 years agoGMod Open Source19 years agoPSP Problems19 years agoMore FW Animation19 years agoSnakes19 years agoFacewound Update19 years agoLost19 years agoSuperman19 years agoGMod Text Corruption19 years agoGMod 9.0.1 released19 years agoGMod 9 released19 years agoquestion_answer 3Gmod 9 Release Date Update19 years agoBlack & White 219 years agoquestion_answer 1More 8.4 gamemodes19 years agoLUA (the programming language)19 years agoquestion_answer 3Dee Dee Bam21 years agoSteam Launched21 years agoquestion_answer 3Driving Test21 years agoquestion_answer 1Updated Game Test22 years agoquestion_answer 1Driving lesson and NS22 years agoquestion_answer 1Poor Titch22 years agoquestion_answer 5
Amazon Cloud Search
13 Years Ago, 2,266 views

Facepunch forums are pretty huge. There’s over 20 million posts in the database. This is cool, but the database regularly locks up when searching. There’s a number of ways to try to get around this, better database server, faster hard drives, pruning posts, slave databases. But whatever you do sooner or later as your post count increases you’re going to have to do it again.

So last week I looked into Amazon Cloudsearch. Put simply you throw all your posts at this service and it provides an API for you to search them. This means that search isn’t using your databases anymore, so your database doesn’t need to be half as powerful, and doesn’t keep locking up.

Inserting Posts

First of all let me start by saying that you can put anything in here. It doesn’t have to be plain text – you can throw pretty much any document at it and it’ll make it searchable.

The way I do it on our forum is simple. I added a ‘indexed’ field to the posts table. When a post is created or edited it sets the ‘indexed’ field to 0.

Then I set up a cron that scan the posts table every minute, then any posts that are indexed=0 it uploads them to CloudSearch. You upload to CloudSearch by sending AJAX queries – it couldn’t be easier. Here’s how I build mine

So obviously I’ve selected the appropriate posts from the database, then I loop through them all adding them to an array. I don’t add them if they are lower than 10 letters, and I strip UTF from the posts to save space (you don’t have to do this, I just decided it’d be a waste of time preserving these characters since we’re an english forum).

Then it’s just a case of converting the array to JSON

And sending that baby to Amazon

And if it succeeds, mark them as indexed..

The Fields

You might have noticed that I send date, forumid, threadid and userid with the posts. This allows us to also search via those fields, and filter by those fields. So if you only want to find threads containing the word ‘Butt’ in General Discussion, posted last week, by me – you can do that easily.

But more than that. When you search it will also categorize your results. It’ll show you the top x forumid’s with that word in, and the top userid’s, and the top threadids.

This allows you to show the results in a way that lets people drill down to find what they want. For example, this search:

http://api.facepunch.com/#/page/1/forum//thread//user//search/max{a1da5f31666ba9e4b613f74c475fa44930e0dd96d95a00cdfc356ce4f15804bc}20payne

You can see that it’s most mentioned in the Max Payne thread. And most mentioned by “A Big Fat Ass”. But it’s also mentioned in the Max Payne 3 thread a lot, and maybe that’s what we’re looking for. So clicking on the thread restricts the results to that thread.

http://api.facepunch.com/#/page/1/forum//thread/1125222/user//search/max{a1da5f31666ba9e4b613f74c475fa44930e0dd96d95a00cdfc356ce4f15804bc}20payne

Not a super example of why this is cool, but compare that process to VBulletin’s search:

Searching shouldn’t be THIS much work.

Getting Results

Searching is as easy as opening a URL. Because that’s all it is. When you create your search domain you’ll be given a unique URL to query. You can do this from inside your site (I wrap it in an API) – or you could query it directly via an ajax request. Their dashboard lets you run test queries, so you can make sure it’s working.

Price

This is where the usefulness will probably drop out for most people. The price starts at around $80 a month. So if you’re providing search for one small thing – it probably wouldn’t be worth it.

It scales with usage. So if you have a hundred million entries, and you’re querying 1000 times a second – it’s gonna cost a lot more $80. But on the upside performance will remain the same.

Right now we have nearly 3 million posts indexed and we’re still on a single small search type (it scales automatically). I am expecting this to change to the large type soon (which is around $350 a month).

Summary

For us – even at $350 a month – I think it’s worth it, for these three reasons.

  • Search Results are Instant
  • Search Results are Better
  • Takes pressure off the Database
question_answer

Add a Comment

An error has occurred. This application may no longer respond until reloaded. Reload 🗙
🔌 Reconnecting..