cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

hasinketi48
by New Contributor
  • 983 Views
  • 0 replies
  • 0 kudos

How is Databricks Spark different than Spark?

Hey guys, I am looking to create a real-time analytics application and I am pretty new to Data engineering. Any advice here would be appreciated. So I have been l appvalleyooking into spark streaming for my transformation process, so th tutuappe ove...

  • 983 Views
  • 0 replies
  • 0 kudos
lawregill92
by New Contributor
  • 1210 Views
  • 1 replies
  • 0 kudos

Question About Access and Filter Data in Databricks

Hi guys, im new using databricks and i have a challenge in my new work. routerlogin I need to access to one the database (the database is on DBFS) result of some ETLS trough any service, can be ODBC or by some API. I need to connect there because I...

  • 1210 Views
  • 1 replies
  • 0 kudos
Latest Reply
Dan_Z
Databricks Employee
  • 0 kudos

Use the Simba ODBC connector: https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html

  • 0 kudos
Skier
by New Contributor
  • 3974 Views
  • 1 replies
  • 1 kudos

Multiple Clusters stuck in pending state during creation.

I have been trying to create a new cluster to use and multiple attempts have gotten stuck in pending: "Finding instances for new nodes, acquiring more instances if necessary" until they time out. Up to today I have had no problems creating clusters ...

  • 3974 Views
  • 1 replies
  • 1 kudos
Latest Reply
Dan_Z
Databricks Employee
  • 1 kudos

This is typically a cloud provider issue. You can file a support ticket if the issue persists.

  • 1 kudos
maffeenAF
by New Contributor
  • 1721 Views
  • 1 replies
  • 0 kudos

How do I make approxSimilarityJoin work on 25k 300-d vectors?

I’m trying to use LSH approxSimilarityJoin on a dataset with ~25k 300-d vectors of floats. It gets stuck and eventually fails with ’Slave lost’ error. The size of cluster and memory are likely not a problem, the failure happens even with 16 nodes, 1...

  • 1721 Views
  • 1 replies
  • 0 kudos
Latest Reply
Dan_Z
Databricks Employee
  • 0 kudos

Use a PandasUDF with Arrow enabled. They are improved in Spark 3, but you can use them in Spark 2.4.5.

  • 0 kudos
davidmory38
by New Contributor
  • 3239 Views
  • 1 replies
  • 0 kudos

Best Database for facial recognition/ Fast comparisons of Euclidean distance

Hello people,I'm trying to build a facial recognition application, and I have a working API, that takes in an image of a face and spits out a vector that encodes it. I need to run this on a million faces, store them in a db and when the system goes o...

  • 3239 Views
  • 1 replies
  • 0 kudos
Latest Reply
Dan_Z
Databricks Employee
  • 0 kudos

You could do this with Spark storing in parquet/Delta. For each face you would write out a record with a column for metadata, a column for the encoded vector array, and other columns for hashing. You could use a PandasUDF to do the distributed dista...

  • 0 kudos
austiamel47
by New Contributor
  • 1370 Views
  • 1 replies
  • 0 kudos

Databricks delta lake

Can we use databricks delta lake as a data warehouse kind of thing where business analysts can explore data according to their needs ? Delta lake provides following features which I think supports this idea support to sql syntaxprovide ACID guarante...

  • 1370 Views
  • 1 replies
  • 0 kudos
Latest Reply
Dan_Z
Databricks Employee
  • 0 kudos

@austiamel47, Yes, you can certainly do this. Delta Lake is designed to be competitive with traditional data warehouses and with some tuning can power low-latency dashboards.https://databricks.com/glossary/data-lakehouse

  • 0 kudos
Ryan_Chynoweth
by Databricks Employee
  • 3071 Views
  • 1 replies
  • 2 kudos
  • 3071 Views
  • 1 replies
  • 2 kudos
Latest Reply
Ryan_Chynoweth
Databricks Employee
  • 2 kudos

Yes. A new workspace would need to be deployed because Azure allows people to change the vnet cidr but it requires you to remove all the vnet resources first. This includes the Databricks deployment, therefore, this is an Azure restriction on how VNE...

  • 2 kudos
wallystart
by New Contributor III
  • 1699 Views
  • 0 replies
  • 1 kudos

Is possible to use jupyter extensions in databricks?

Hi, we need create an interactive map from ipyleaflet library and this use jupyterlab extensionjupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter-leafletWe achieved to show with displayHTML but we lose the widget events

  • 1699 Views
  • 0 replies
  • 1 kudos
jholder
by New Contributor II
  • 2253 Views
  • 2 replies
  • 1 kudos

Cluster Pending

Hello, Relatively new to Databricks and I've been using the Community Edition for a little bit now. I've recently been having more and more issues with my clusters pending until they time out before ever starting up. I've seen a few other posts here...

  • 2253 Views
  • 2 replies
  • 1 kudos
Latest Reply
GustavoRocha
New Contributor III
  • 1 kudos

It seems that it's working now. At least for me...

  • 1 kudos
1 More Replies
daniil_terentye
by New Contributor III
  • 1359 Views
  • 1 replies
  • 0 kudos

New task orchestration tool

Hi everybody! Is it possible to use one job cluster for multiple tasks using new task orchestration tool? If so then please tell how. If it's impossible then this new tool looks useless Regards, Daniil.

  • 1359 Views
  • 1 replies
  • 0 kudos
Latest Reply
daniil_terentye
New Contributor III
  • 0 kudos

Databricks team, are there any news about it?

  • 0 kudos
MichaelBlahay
by New Contributor
  • 3327 Views
  • 1 replies
  • 1 kudos

Where is dbfs mounted with community edition?

The the regular version of databricks, the dbfs is mounted at /dbfs. This does not seem to be the case with community edition. I am seeking more details.

  • 3327 Views
  • 1 replies
  • 1 kudos
Latest Reply
AlexandrePetrul
New Contributor II
  • 1 kudos

If you are using DBR 7.x. or newer versions the dbfs is disabled. You have to use dbutils.fs.cp commands as a workaround.

  • 1 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels