cancel
Showing results for 
Search instead for 
Did you mean: 
Page Title

Welcome to the Databricks Community

Discover the latest insights, collaborate with peers, get help from experts and make meaningful connections

101209members
51101posts
cancel
Showing results for 
Search instead for 
Did you mean: 
DigiCert is updating its root CA certificate

Databricks uses DigiCert as its SSL certificate authority, to establish trust. DigiCert is making a change to its root and intermediate CA certificates (detailed information below). This change will impact Databricks, starting on May 15, 2024. If you...

  • 313 Views
  • 1 replies
  • 0 kudos
Friday
Big news: Our Community is now 100,000 members strong with over 50,000 posts🚀

Thanks to every one of you, the Databricks Community has reached an incredible milestone: 100,000 members and over 50,000 posts! Your dedication, expertise and passion have made this possible. Whether you're a seasoned data professional, a coding en...

  • 2191 Views
  • 8 replies
  • 7 kudos
3 weeks ago
LIMIT: Less is More for Instruction Tuning

Ready to elevate your language model's performance in question answering? Embrace the power of supervised fine-tuning on a small set of high-quality samples. Discover how a few thousand carefully selected training pairs can enhance your model's capab...

  • 417 Views
  • 0 replies
  • 2 kudos
Wednesday
Featured Member Interview - March 2024 - Janga Phanindra Reddy

Before delving into the interview, meet Janga Reddy, an Associate Principal at LTIMindtree with a passion for data engineering and a plethora of Databricks certifications. We posed several questions, and here are his answers, in his own words. Name: ...

  • 2048 Views
  • 4 replies
  • 5 kudos
3 weeks ago

Community Activity

pernilak
by New Contributor II
  • 1 Views
  • 0 replies
  • 0 kudos

Best practices for working with external locations where many files arrive constantly

I have an Azure Function that receives files (not volumes) and dumps them to cloud storage. One-five files are received approx. per second. I want to create a partitioned table in Databricks to work with. How should I do this? E.g.: register the cont...

  • 1 Views
  • 0 replies
  • 0 kudos
Pragati_17
by New Contributor
  • 10 Views
  • 1 replies
  • 0 kudos

Setting Date Range Picker to some default date

I have a date range picker filter in Databricks Lakeview Dashboard, so when i open dashboard there is no date selected and i want to set a default date. Is that possible with lakeview dashboard filters?

Data Engineering
databricks lakeview dashboard
date range pciker filter
default date set
  • 10 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Pragati_17, You can set a default date for your date range picker filter in Databricks Lakeview Dashboards. Here’s how: 1. Create a Lakeview Dashboard: - Click New in the sidebar and select Lakeview Dashboard.- Your new dashboard is automatica...

  • 0 kudos
Gilg
by Contributor
  • 17 Views
  • 0 replies
  • 0 kudos

Autoloader - File Notification mode

Hi All,I have set up a DLT pipeline that is using Autoloader in a file notification mode.Everything runs smoothly for the first time. However, it seems like the next micro-batch did not trigger as I can see some events coming in the queue.But if I lo...

Gilg_0-1710827649089.png Gilg_1-1710827662118.png
  • 17 Views
  • 0 replies
  • 0 kudos
pranathisg97
by New Contributor III
  • 635 Views
  • 3 replies
  • 1 kudos

Resolved! Control query caching using SQL statement execution API

I want to execute this statement using databricks SQL Statement Execution API. curl -X POST -H 'Authorization: Bearer <access-token>' -H 'Content-Type: application/json' -d '{"warehouse_id": "<warehouse_id>", "statement": "set us...

image.png
  • 635 Views
  • 3 replies
  • 1 kudos
Latest Reply
TimFrazer
New Contributor II
  • 1 kudos

Did you ever find a solution to this problem?

  • 1 kudos
2 More Replies
Brichj
by New Contributor
  • 54 Views
  • 2 replies
  • 0 kudos

%run ../Includes/Classroom-Setup-02.1

I ran the code in the cell as it was given in the presentation. But it failed. Can someone please help?The presentation is the second lesson in the second model of Data Engineering Associate exam prep.

  • 54 Views
  • 2 replies
  • 0 kudos
Latest Reply
Brichj
New Contributor
  • 0 kudos

Thanks Ajay-Pandey!This is error that I keep getting when I run the following: %run ./Includes/Classroom-Setup-02.3LI have run dbutils.library.restartPython(), but it did not help.Note: you may need to restart the kernel using dbutils.library.restart...

  • 0 kudos
1 More Replies
Brad
by New Contributor III
  • 220 Views
  • 4 replies
  • 0 kudos

Inconsistent behavior when displaying chart in notebook

Hi, I'm trying to create some 3D charts. With the same code and same cluster, sometimes it can show, sometimes it cannot. Previously it cannot display, but last week I opened a notebook with failed run and found the result can be shown by itself (as ...

  • 220 Views
  • 4 replies
  • 0 kudos
Latest Reply
Brad
New Contributor III
  • 0 kudos

Also, with same code, same browser, different workspaces, one works, other one not. In the notebook with "script error", if I "Export cell" and get its iframe html and use displayHTML to display it, it works, so this means the JS and HTML inside is o...

  • 0 kudos
3 More Replies
Sampath_Kumar
by New Contributor
  • 81 Views
  • 2 replies
  • 1 kudos

Volume Limitations

I have a use case to create a table using JSON files. There are 36 million files in the upstream(S3 bucket). I just created a volume on top of it. So the volume has 36M files.  I'm trying to form a data frame by reading this volume using the below sp...

  • 81 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @Sampath_Kumar, Let’s delve into the limitations and best practices related to Databricks volumes. Volume Limitations: Managed Volumes: These are Unity Catalog-governed storage volumes created within the default storage location of the contain...

  • 1 kudos
1 More Replies
sanjay
by Valued Contributor II
  • 656 Views
  • 12 replies
  • 3 kudos

Performance issue while calling mlflow endpoint

Hi,I have pyspark dataframe and pyspark udf which calls mlflow model for each row but its performance is too slow.Here is sample codedef myfunc(input_text):   restult = mlflowmodel.predict(input_text)   return resultmyfuncUDF = udf(myfunc,StringType(...

  • 656 Views
  • 12 replies
  • 3 kudos
Latest Reply
Isabeente
New Contributor II
  • 3 kudos

So good

  • 3 kudos
11 More Replies
cpd
by Visitor
  • 33 Views
  • 0 replies
  • 0 kudos

Ingesting geospatial data into a table

I'm just getting started with Databricks and wondering if it is possible to ingest a GeoJSON or GeoParquet file into a new table without writing code? My goal here is to load vector data into a table and perform H3 polyfill operations on all the vect...

  • 33 Views
  • 0 replies
  • 0 kudos
xhead
by New Contributor II
  • 1702 Views
  • 3 replies
  • 0 kudos

Resolved! Does "databricks bundle deploy" clean up old files?

I'm looking at this page (Databricks Asset Bundles development work tasks) in the Databricks documentation.When repo assets are deployed to a databricks workspace, it is not clear if the "databricks bundle deploy" will remove files from the target wo...

Data Engineering
bundle
cli
deploy
  • 1702 Views
  • 3 replies
  • 0 kudos
Latest Reply
xhead
New Contributor II
  • 0 kudos

One further question:The purpose of “databricks bundle destroy” is to remove all previously-deployed jobs, pipelines, and artifacts that are defined in the bundle configuration files.Which bundle configuration files? The ones in the repo? Or are ther...

  • 0 kudos
2 More Replies
johnp
by New Contributor
  • 173 Views
  • 0 replies
  • 0 kudos

pdb debugger on databricks

I am new to databricks. and trying to debug my python application with variable-explore by following the instruction from: https://www.databricks.com/blog/new-debugging-features-databricks-notebooks-variable-explorerI added the "import pdb" in the fi...

  • 173 Views
  • 0 replies
  • 0 kudos
Ramakrishnan83
by New Contributor III
  • 188 Views
  • 1 replies
  • 0 kudos

Understanding Spark Architecture during Table Creation

Team ,I am trying understand how the parquet files and JSON under the delta log folder stores the data behind the scenesTable Creation:from delta.tables import *DeltaTable.create(spark) \.tableName("employee") \.addColumn("id", "INT") \.addColumn("na...

Ramakrishnan83_0-1710772217666.png Ramakrishnan83_1-1710772318911.png Ramakrishnan83_2-1710772374126.png
  • 188 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Honored Contributor III
  • 0 kudos

@Ramakrishnan83  - Kindly go through the blog post - https://www.databricks.com/blog/2019/08/21/diving-into-delta-lake-unpacking-the-transaction-log.html which discuss in detail on delta's transaction log.

  • 0 kudos
pokus
by New Contributor III
  • 2084 Views
  • 3 replies
  • 2 kudos

Resolved! use DeltaLog class in databricks cluster

I need to use DeltaLog class in the code to get the AddFiles dataset. I have to keep the implemented code in a repo and run it in databricks cluster. Some docs say to use org.apache.spark.sql.delta.DeltaLog class, but it seems databricks gets rid of ...

  • 2084 Views
  • 3 replies
  • 2 kudos
Latest Reply
dbal
New Contributor
  • 2 kudos

Thanks for providing a solution @pokus .What I dont understand is why Databricks cannot provide the DeltaLog at runtime. How can this be the official solution? We need a better solution for this instead of depending on reflections.

  • 2 kudos
2 More Replies
VGS777
by New Contributor II
  • 45 Views
  • 0 replies
  • 0 kudos

Regarding Cloning dbc file from git

Hey FolksI have dbc file in a git repo and i cloned in the databricks when tried to open the .dbc file it is saying ```Failed to load file. The file encoding is not supported```can anyone please advice me on this #help #beginner

  • 45 Views
  • 0 replies
  • 0 kudos
Learnit
by New Contributor II
  • 29 Views
  • 0 replies
  • 0 kudos

Delta Sharing resulting in Bad_Request

Hi All,Recipeint encountering an issue while trying to access my organizational data (providers data) in a Delta Sharing scenario(databricks to databricks), and I'm hoping to get some guidance on how to resolve it. Here is the error message recipient...

  • 29 Views
  • 0 replies
  • 0 kudos

Latest from our Blog

IDENTIFIER: Turning ticks to backticks since DBR 13.2

Motivation Note: You can find all examples to run here. In past posts, we discussed parameter markers that you can use to templatize queries. Given a simple example table: CREATE OR REPLACE TABLE resi...

738Views 1kudos