cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

cconnell
by Contributor II
  • 583 Views
  • 1 replies
  • 2 kudos

www.linkedin.com

Here is an article I wrote that puts Databricks in a historical context (why was it developed?) and provides introductory steps to help a newbie get started. Feel free to copy/link as you want.https://www.linkedin.com/pulse/databricks-introduction-ch...

  • 583 Views
  • 1 replies
  • 2 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Chuck Connell​ , Thank you for posting such an informative article!

  • 2 kudos
cconnell
by Contributor II
  • 684 Views
  • 2 replies
  • 1 kudos

www.linkedin.com

Importing JSON to Databricks (PySpark) is simple in the simple case. But of course there are wrinkles for real-world data. Here are some tips/tricks to help...https://www.linkedin.com/pulse/json-databricks-pyspark-chuck-connell/

  • 684 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Chuck Connell​ , Thank you for sharing such an amazing article!

  • 1 kudos
1 More Replies
William_Scardua
by Valued Contributor
  • 2798 Views
  • 4 replies
  • 3 kudos

Resolved! Update Databricks Runtime

Hy guys,I need to upgrade my databricks runtime (current 8.0 ) What the precautions should I take ?Thank you very much

  • 2798 Views
  • 4 replies
  • 3 kudos
Latest Reply
Steward475
New Contributor II
  • 3 kudos

If you want to know the version of Databricks runtime in Azure after creation: Go to Azure Data bricks portal => Clusters => Interactive Clusters => here you can find the run time version. For more details, refer "Azure Databricks Runtime versions".R...

  • 3 kudos
3 More Replies
test_data
by New Contributor III
  • 4006 Views
  • 2 replies
  • 1 kudos
  • 4006 Views
  • 2 replies
  • 1 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

Please use Repos and in admin settings please enable "Files in Repo" than you will be able to import class in notebook:from repo_folder.subfolders.file import your_class

  • 1 kudos
1 More Replies
test_data
by New Contributor III
  • 3693 Views
  • 4 replies
  • 2 kudos

need to move notebook file from workspace to dbfs.

Hi teami need to move the notebook file from workspace to dbfs. i have tried and getting an error there is no file ?

  • 3693 Views
  • 4 replies
  • 2 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

Not sure what command are you using and what is error? As access to workspace is managed per user on databricks so %sh magic commands will not work as you could see other user files.Dbfs is data storage. Notebook code from Workspace can be moved to R...

  • 2 kudos
3 More Replies
prasadvaze
by Valued Contributor II
  • 1003 Views
  • 1 replies
  • 1 kudos

which table in external hive metastore stores the folder path for delta table's data?

which table in external hive metastore stores the folder path for delta table's data? Is it SDS table?

  • 1003 Views
  • 1 replies
  • 1 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

Image here can be helpful https://analyticsanvil.files.wordpress.com/2016/08/hive_metastore_database_diagram.png

  • 1 kudos
BorislavBlagoev
by Valued Contributor III
  • 3307 Views
  • 11 replies
  • 2 kudos

Resolved! Databricks SQL Connector

I started to explore Databricks SQL Connector. And I'm wondering can I execute query from Databricks SQL Connector from notebook or job?sample code:'test_query' lives in Databricks SQL. It is ready to use query!from databricks import sql   hostname =...

  • 3307 Views
  • 11 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Borislav Blagoev​ - There is a "Select as Best" link underneath each reply. All you need to do, if you choose, is to click on that link for whichever reply best answered your question. The purpose is to help other members find helpful solutions more...

  • 2 kudos
10 More Replies
ray21
by New Contributor II
  • 1964 Views
  • 4 replies
  • 2 kudos

Do Databricks support XLA compilation for TensorFlow models?

I am defining a sequential Keras model using tensorflow.kerasRuntime: Databricks ML 8.3Cluster: Standard NC24 with 4 GPUs per node.To enable XLA compilation, I set the following flag:tf.config.optimizer.set_jit(True)Here is the output when I try to t...

  • 1964 Views
  • 4 replies
  • 2 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 2 kudos

I don't think this is specific to Databricks, but rather Tensorflow. See https://stackoverflow.com/questions/68614547/tensorflow-libdevice-not-found-why-is-it-not-found-in-the-searched-path for a possibly relevant solution.I don't see evidence that t...

  • 2 kudos
3 More Replies
Atul_Sharan
by New Contributor II
  • 3420 Views
  • 5 replies
  • 2 kudos

Resolved! Hello Experts - I am facing one technical issue with Databricks SQL - IF-ELSE or CASE statement implementation when trying to execute two separate set of queries based on a valued of a column of the Delta table.

Hi Experts,I'm reading a pipe delimited source file where first row does not contain the data but contains the - REPLACE or UPDATE values which indicate the refresh type if it is Full refresh Or Upsert. The second row is nothing but header and the a...

  • 3420 Views
  • 5 replies
  • 2 kudos
Latest Reply
Atul_Sharan
New Contributor II
  • 2 kudos

Thanks a ton Gurpreet, Suggested approach was helpful in resolution of the issue. Appreciate your help !!

  • 2 kudos
4 More Replies
Orianh
by Valued Contributor II
  • 2930 Views
  • 5 replies
  • 1 kudos

Train deep learning model with numpy arrays.

Hey guys,I'm trying to train deep learning model at ML databricks with numpy arrays as input.For now i organized all the data inside DF- df contains 4 columns : col1,col2,col3,col4col1 and col2 have arrays with shape (1,3,3,3,3), col 3 have array wit...

  • 2930 Views
  • 5 replies
  • 1 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

Maybe you could save some your code. It will be easier to answer and also we could learn deep learning in databricks from your code.

  • 1 kudos
4 More Replies
DamienSicard
by New Contributor III
  • 6686 Views
  • 4 replies
  • 2 kudos

Resolved! Notebooks font size

Hi,Is there a way to increase the cells' font size and set it as a default setting ?Thanks.Best Damien

  • 6686 Views
  • 4 replies
  • 2 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Damien Sicard​ , As @werners has stated, you can zoom your browser.

  • 2 kudos
3 More Replies
herry
by New Contributor III
  • 4693 Views
  • 5 replies
  • 1 kudos

Hive Serde table vs Delta table

This might be stupid question. Does the Hive Serde table have the same features (e.g. transactions) comparing to the Delta table?I tried to find the information in the Databricks documentation but I cannot find a clear answer.I create the Hive Serde ...

  • 4693 Views
  • 5 replies
  • 1 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

AFAIK Hive SerDe is just Serializer and Deserializer (write and read data to/from storage).Hive uses SerDe (and FileFormat) to read and write table rows. So it is not an actual file format like parquet, orc and also delta lake (which I consider a se...

  • 1 kudos
4 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels