cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Science & Machine Learning

Forum Posts

kll
by New Contributor III
  • 6204 Views
  • 1 replies
  • 1 kudos

Internal error: com.databricks.rpc.RPCResponseTooLarge, when attempting to use mosaic's st_intersects

I get an exception when attempting to run the following line of code, which filters a spark DataFrame based on the geometry.df_tx = df_zip.filter(st_intersects(st_aswkt("zip_code_geom"), tx_poly))   df_tx.show()where, `tx_poly` is,`tx_poly = shapely....

  • 6204 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

I am not familiar with st_intersects, but it seems that it runs solely on the driver (as python code, not spark).Does mosaic work in pyspark?If not: try to use a larger driver.

  • 1 kudos
mbaumga
by New Contributor III
  • 3375 Views
  • 7 replies
  • 9 kudos

How to request the addition of pre-installed R packages on the clusters?

Today, many R packages are pre-installed on the standard clusters on Databricks. Libraries like "tidyverse", "ggplot2", etc are there. Also the great library "readxl" to load Excel files. But unfortunately, its counterpart "writexl" is not pre-instal...

  • 3375 Views
  • 7 replies
  • 9 kudos
Latest Reply
wicckkjoe
New Contributor II
  • 9 kudos

I just need to figure who decides which R packages are pre-installed on the cluster.

  • 9 kudos
6 More Replies
sher
by Valued Contributor II
  • 1249 Views
  • 1 replies
  • 1 kudos

how to execute the CHANGE_TRACKING snowflake query in databricks

please check the attached image which needs to resolve. anyone have come across this kind of issues

image (1)
  • 1249 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @sherbin w​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
rahullalwani25
by New Contributor II
  • 6429 Views
  • 3 replies
  • 4 kudos

to_utc_timestamp is subtracting a different time delta and from_utc_timestamp is not adding the same delta.

My session timezone is Australia/Sydneyif i run the below query my expectation is first column and third column should show the same value. But it is not working as expected for 1753-01-01 00:00:00 timestamp.spark.conf.set("spark.sql.session.timeZone...

image.png
  • 6429 Views
  • 3 replies
  • 4 kudos
Latest Reply
Pavithra_R
Databricks Employee
  • 4 kudos

Hi @Rahul Lalwani​  (Customer)​,In Interactive cluster spark.sql.datetime.java8API.enabled is disabled when we enable spark.sql.datetime.java8API.enabled to true , we can see crt values for 1753-01-01 as well.The reason for enabling the above config ...

  • 4 kudos
2 More Replies
Mado
by Valued Contributor II
  • 5145 Views
  • 2 replies
  • 1 kudos

Error "Invalid configuration value detected for fs.azure.account.key" when listing files stored in an Azure Storage account using "dbutils.fs.ls"

I get the following error when getting a list of files stored in an Azure Storage account using "dbutils.fs.ls" command in Databrciks.Failure to initialize configuration for storage account AAAAA.dfs.core.windows.net: Invalid configuration value dete...

0 1 2 6
  • 5145 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Mohammad Saber​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
1 More Replies
marvin1
by New Contributor III
  • 431 Views
  • 1 replies
  • 0 kudos

Unable to configure compute of cloned job

If I clone an existing job without making any changes, I am able to reconfigure the compute successfully. If I remove or add a spark environment variable to test modifications, such as using secrets for example, and I confirm the changes to the job, ...

  • 431 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Marvin Ginns​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
david_torres
by New Contributor II
  • 2726 Views
  • 3 replies
  • 4 kudos

Can you use autoloader with a fixed width file?

I have a collection fixed width files that I would like to ingest monthly with autoloader but I can't seem to find an example. I can read the files into Dataframes using a python function to map the index and length of each field with no issues but ...

  • 2726 Views
  • 3 replies
  • 4 kudos
Latest Reply
david_torres
New Contributor II
  • 4 kudos

I found a way to get what I needed and I can apply this to any fixed width file. Will share for anyone trying to do the same thing. I accomplished this in a Python notebook and will explain the code:Import the libraries needed and define a schema.i...

  • 4 kudos
2 More Replies
g96g
by New Contributor III
  • 905 Views
  • 1 replies
  • 0 kudos

Read the file from datalake in databricks -No such file directory error

0I have a problem with reading the file from ADLS gen 2.I have dont the mounting properly as after executing dbutils.fs.ls('/mnt/bronze') I can see the file path.the way how I did the mounting:  # dbutils.fs.mount( # source = "abfss://"+container_r...

  • 905 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Givi Salu​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
alesventus
by Contributor
  • 2583 Views
  • 1 replies
  • 2 kudos

Pyspark Merge parquet and delta file

Is it possible to use merge command when source file is parquet and destination file is delta? Or both files must delta files? Currently, I'm using this code and I transform parquet into delta and it works. But I want to avoid of this tranformation.T...

  • 2583 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Ales ventus​ We haven't heard from you since the last response from @Kaniz Fatma​ , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others...

  • 2 kudos
Anonymous
by Not applicable
  • 601 Views
  • 0 replies
  • 0 kudos

 HI ML Practitioners,  I want to ask you all how are you productionizing your ML workloads? Are you using ML Flow?Whats your take on ML Flow Recipies?...

 HI ML Practitioners, I want to ask you all how are you productionizing your ML workloads? Are you using ML Flow?Whats your take on ML Flow Recipies? Lets get the conversation started.MLflow Recipes (previously known as MLflow Pipelines) is a framewo...

Copy of 2023-05-Community-ongoing-announcement-1200x628
  • 601 Views
  • 0 replies
  • 0 kudos
PGrover
by New Contributor II
  • 1461 Views
  • 1 replies
  • 2 kudos

Connecting to Synapse database using AzureCliCredential token in Spark

I want to connect to my Azure Synapse database using Spark. I can do this in pyodbc no problem but that is not what I want.Here is how I get my credentialscredential = AzureCliCredential() databaseToken = credential.get_token('https://database.window...

  • 1461 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Patrick Grover​ We haven't heard from you since the last response from @Kaniz Fatma​ â€‹, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to ot...

  • 2 kudos
Soma
by Valued Contributor
  • 1751 Views
  • 2 replies
  • 3 kudos

View rocksdb statestore

I used rocksdb statestore for streaming and use collect_set to achieve de-dup logic.Is there any way I can convert rocksdb key value iterator to normal string as I need to validate how internally it is stored as I might need to store 50k distinct val...

  • 1751 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @somanath Sankaran​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 3 kudos
1 More Replies
VVM
by New Contributor III
  • 1342 Views
  • 2 replies
  • 2 kudos

Bug with display function with structs?

It appears to me that there's a deceptive bug when using the databricks display function and viewing struct data. For whatever reason, multiple spaces are cut down to only one:from pyspark.sql.functions import struct, col   df = spark.createDataFrame...

  • 1342 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Patrick Mascari​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 2 kudos
1 More Replies
ChrisS
by New Contributor III
  • 10453 Views
  • 14 replies
  • 5 kudos

Resolved! How do you get data from Azure Data Lake Gen 2 Mounted or Imported and Exported from Databricks?

The example that data bricks gives is not helpful and does not tell me exactly what I need to do. I am new to this and not sure what I need to do in azure to get this done. I just need to be able to pull data and write data to the data containers. Be...

  • 10453 Views
  • 14 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Chris Sarrico​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 5 kudos
13 More Replies
Priyag1
by Honored Contributor II
  • 1821 Views
  • 1 replies
  • 5 kudos

docs.databricks.com

New unified Databricks navigationDatabricks plans to enable the new navigation experience (Public Preview) by default for all users. You’ll be able to opt out by clicking Disable new UI in the sidebar.The goal of the new experience is to reduce click...

  • 1821 Views
  • 1 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Priyadarshini G​ Thank you for providing accurate and valuable information.Best Regards

  • 5 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels