cancel
Showing results for 
Search instead for 
Did you mean: 
Page Title

Welcome to the Databricks Community

Discover the latest insights, collaborate with peers, get help from experts and make meaningful connections

101198members
51096posts
cancel
Showing results for 
Search instead for 
Did you mean: 
DigiCert is updating its root CA certificate

Databricks uses DigiCert as its SSL certificate authority, to establish trust. DigiCert is making a change to its root and intermediate CA certificates (detailed information below). This change will impact Databricks, starting on May 15, 2024. If you...

  • 305 Views
  • 1 replies
  • 0 kudos
Friday
Big news: Our Community is now 100,000 members strong with over 50,000 posts🚀

Thanks to every one of you, the Databricks Community has reached an incredible milestone: 100,000 members and over 50,000 posts! Your dedication, expertise and passion have made this possible. Whether you're a seasoned data professional, a coding en...

  • 2175 Views
  • 8 replies
  • 7 kudos
3 weeks ago
LIMIT: Less is More for Instruction Tuning

Ready to elevate your language model's performance in question answering? Embrace the power of supervised fine-tuning on a small set of high-quality samples. Discover how a few thousand carefully selected training pairs can enhance your model's capab...

  • 413 Views
  • 0 replies
  • 2 kudos
Wednesday
Featured Member Interview - March 2024 - Janga Phanindra Reddy

Before delving into the interview, meet Janga Reddy, an Associate Principal at LTIMindtree with a passion for data engineering and a plethora of Databricks certifications. We posed several questions, and here are his answers, in his own words. Name: ...

  • 2044 Views
  • 4 replies
  • 5 kudos
3 weeks ago

Community Activity

Brichj
by New Contributor
  • 50 Views
  • 2 replies
  • 0 kudos

%run ../Includes/Classroom-Setup-02.1

I ran the code in the cell as it was given in the presentation. But it failed. Can someone please help?The presentation is the second lesson in the second model of Data Engineering Associate exam prep.

  • 50 Views
  • 2 replies
  • 0 kudos
Latest Reply
Brichj
New Contributor
  • 0 kudos

Thanks Ajay-Pandey!This is error that I keep getting when I run the following: %run ./Includes/Classroom-Setup-02.3LI have run dbutils.library.restartPython(), but it did not help.Note: you may need to restart the kernel using dbutils.library.restart...

  • 0 kudos
1 More Replies
Brad
by New Contributor II
  • 76 Views
  • 4 replies
  • 0 kudos

Inconsistent behavior when displaying chart in notebook

Hi, I'm trying to create some 3D charts. With the same code and same cluster, sometimes it can show, sometimes it cannot. Previously it cannot display, but last week I opened a notebook with failed run and found the result can be shown by itself (as ...

  • 76 Views
  • 4 replies
  • 0 kudos
Latest Reply
Brad
New Contributor II
  • 0 kudos

Also, with same code, same browser, different workspaces, one works, other one not. In the notebook with "script error", if I "Export cell" and get its iframe html and use displayHTML to display it, it works, so this means the JS and HTML inside is o...

  • 0 kudos
3 More Replies
Sampath_Kumar
by New Contributor
  • 78 Views
  • 2 replies
  • 1 kudos

Volume Limitations

I have a use case to create a table using JSON files. There are 36 million files in the upstream(S3 bucket). I just created a volume on top of it. So the volume has 36M files.  I'm trying to form a data frame by reading this volume using the below sp...

  • 78 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @Sampath_Kumar, Let’s delve into the limitations and best practices related to Databricks volumes. Volume Limitations: Managed Volumes: These are Unity Catalog-governed storage volumes created within the default storage location of the contain...

  • 1 kudos
1 More Replies
sanjay
by Valued Contributor II
  • 650 Views
  • 12 replies
  • 3 kudos

Performance issue while calling mlflow endpoint

Hi,I have pyspark dataframe and pyspark udf which calls mlflow model for each row but its performance is too slow.Here is sample codedef myfunc(input_text):   restult = mlflowmodel.predict(input_text)   return resultmyfuncUDF = udf(myfunc,StringType(...

  • 650 Views
  • 12 replies
  • 3 kudos
Latest Reply
Isabeente
New Contributor II
  • 3 kudos

So good

  • 3 kudos
11 More Replies
cpd
by Visitor
  • 20 Views
  • 0 replies
  • 0 kudos

Ingesting geospatial data into a table

I'm just getting started with Databricks and wondering if it is possible to ingest a GeoJSON or GeoParquet file into a new table without writing code? My goal here is to load vector data into a table and perform H3 polyfill operations on all the vect...

  • 20 Views
  • 0 replies
  • 0 kudos
xhead
by New Contributor II
  • 1700 Views
  • 3 replies
  • 0 kudos

Resolved! Does "databricks bundle deploy" clean up old files?

I'm looking at this page (Databricks Asset Bundles development work tasks) in the Databricks documentation.When repo assets are deployed to a databricks workspace, it is not clear if the "databricks bundle deploy" will remove files from the target wo...

Data Engineering
bundle
cli
deploy
  • 1700 Views
  • 3 replies
  • 0 kudos
Latest Reply
xhead
New Contributor II
  • 0 kudos

One further question:The purpose of “databricks bundle destroy” is to remove all previously-deployed jobs, pipelines, and artifacts that are defined in the bundle configuration files.Which bundle configuration files? The ones in the repo? Or are ther...

  • 0 kudos
2 More Replies
johnp
by New Contributor
  • 159 Views
  • 0 replies
  • 0 kudos

pdb debugger on databricks

I am new to databricks. and trying to debug my python application with variable-explore by following the instruction from: https://www.databricks.com/blog/new-debugging-features-databricks-notebooks-variable-explorerI added the "import pdb" in the fi...

  • 159 Views
  • 0 replies
  • 0 kudos
Ramakrishnan83
by New Contributor III
  • 184 Views
  • 1 replies
  • 0 kudos

Understanding Spark Architecture during Table Creation

Team ,I am trying understand how the parquet files and JSON under the delta log folder stores the data behind the scenesTable Creation:from delta.tables import *DeltaTable.create(spark) \.tableName("employee") \.addColumn("id", "INT") \.addColumn("na...

Ramakrishnan83_0-1710772217666.png Ramakrishnan83_1-1710772318911.png Ramakrishnan83_2-1710772374126.png
  • 184 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Honored Contributor III
  • 0 kudos

@Ramakrishnan83  - Kindly go through the blog post - https://www.databricks.com/blog/2019/08/21/diving-into-delta-lake-unpacking-the-transaction-log.html which discuss in detail on delta's transaction log.

  • 0 kudos
pokus
by New Contributor III
  • 2079 Views
  • 3 replies
  • 2 kudos

Resolved! use DeltaLog class in databricks cluster

I need to use DeltaLog class in the code to get the AddFiles dataset. I have to keep the implemented code in a repo and run it in databricks cluster. Some docs say to use org.apache.spark.sql.delta.DeltaLog class, but it seems databricks gets rid of ...

  • 2079 Views
  • 3 replies
  • 2 kudos
Latest Reply
dbal
New Contributor
  • 2 kudos

Thanks for providing a solution @pokus .What I dont understand is why Databricks cannot provide the DeltaLog at runtime. How can this be the official solution? We need a better solution for this instead of depending on reflections.

  • 2 kudos
2 More Replies
VGS777
by New Contributor II
  • 37 Views
  • 0 replies
  • 0 kudos

Regarding Cloning dbc file from git

Hey FolksI have dbc file in a git repo and i cloned in the databricks when tried to open the .dbc file it is saying ```Failed to load file. The file encoding is not supported```can anyone please advice me on this #help #beginner

  • 37 Views
  • 0 replies
  • 0 kudos
Learnit
by New Contributor II
  • 24 Views
  • 0 replies
  • 0 kudos

Delta Sharing resulting in Bad_Request

Hi All,Recipeint encountering an issue while trying to access my organizational data (providers data) in a Delta Sharing scenario(databricks to databricks), and I'm hoping to get some guidance on how to resolve it. Here is the error message recipient...

  • 24 Views
  • 0 replies
  • 0 kudos
RobsonNLPT
by Contributor
  • 102 Views
  • 3 replies
  • 0 kudos

Resolved! scala-xml : how to move child to another parent node

Hi allThe mandatory rowTag for writing to XML cause doesn't make any sense as I have the complete nested dataframe schema.In my case I need to implement an extra step to remove that extra node (default: Row) after xml generation.I need some examples ...

  • 102 Views
  • 3 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @RobsonNLPT, Working with XML in Scala using the scala-xml library can be powerful and flexible. Let’s break down your requirements and provide an example of how to achieve this. Removing the “Row” Node: When converting a DataFrame to XML, th...

  • 0 kudos
2 More Replies
Linglin
by New Contributor
  • 72 Views
  • 2 replies
  • 0 kudos

How to pass multiple Value to a dynamic Variable in Dashboard underlying SQL

select         {{user_defined_variable}} as my_var,                   count(*) as cntfrom            my_tablewhere         {{user_defined_variable}} = {{value}} for user_defined_variable, I use query based dropdown list to get a column_name I'd like ...

  • 72 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Linglin, It seems you’re dealing with user-defined variables in your SQL query, and you want to dynamically set both the column name and the value in your WHERE clause. Let’s break down the solution: Setting User-Defined Variables: You can s...

  • 0 kudos
1 More Replies
LoiNguyen
by New Contributor II
  • 9447 Views
  • 5 replies
  • 2 kudos

The authentication type 10 is not supported

I use below code to connect to postgresql. df = spark.read \ .jdbc("jdbc:postgresql://hostname:5432/dbname", "schema.table", properties={"user": "user", "password": "password"})\ .load() df.printSchema() However, I got the ...

  • 9447 Views
  • 5 replies
  • 2 kudos
Latest Reply
simboss
New Contributor II
  • 2 kudos

But how are we going to do this for those who use Windows?

  • 2 kudos
4 More Replies
m997al
by Contributor
  • 167 Views
  • 0 replies
  • 0 kudos

Azure Databricks with standard private link (only one Databricks authentication workspace)?

We have successfully set up Azure Databricks with standard private link (front-end and back-end).The front-end uses the authentication workspace as prescribed in the documentation.Suppose we use a "custom DNS" for the configuration below.  Can we set...

m997al_0-1710778785048.png
  • 167 Views
  • 0 replies
  • 0 kudos

Latest from our Blog

IDENTIFIER: Turning ticks to backticks since DBR 13.2

Motivation Note: You can find all examples to run here. In past posts, we discussed parameter markers that you can use to templatize queries. Given a simple example table: CREATE OR REPLACE TABLE resi...

731Views 1kudos