cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
shkelzeen
by • New Contributor II
  • 696 Views
  • 2 replies
  • 0 kudos

Databricks JDBC driver multi query in one request.

Can I run multi query in one command using databricks JDBC driver and would databricks execute one query faster then running multi queries in one script?  

Get Started Discussions
Reply
  • 696 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @shkelzeen,  You can run multiple queries in one command using Databricks JDBC driver by using the operator to chain multiple SQL statements together.    This will execute two queries in one command and display the results in separate tables.   ...

  • 0 kudos
1 More Replies
harvey-c
by • New Contributor III
  • 437 Views
  • 1 replies
  • 0 kudos

DLT pipeline access external location with abfss protocol was failed

Dear Databricks Community Members:The symptom: The DLT pipeline was failed with the error message:  Failure to initialize configuration for storage account storageaccount.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account...

Get Started Discussions
Reply
  • 437 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @harvey-c,    One common issue is using the container names instead of the storage accounts in the URL, or including the container names in the configs. Ensure you use the correct storage account name and key in your code and settings.Another comm...

  • 0 kudos
ChristianRRL
by • Contributor II
  • 869 Views
  • 3 replies
  • 1 kudos

Auto Loader Use Case Question - Centralized Dropzone to Bronze?

Good day,I am trying to use Auto Loader (potentially extending into DLT in the future) to easily pull data coming from an external system (currently located in a single location) and organize it and load it respectively. I am struggling quite a bit a...

Get Started Discussions
Reply
  • 869 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
ChristianRRL
Contributor II
  • 1 kudos

Quick follow-up on this @Kaniz (or to anyone else in the Databricks multi-verse who is able to help clarify this case).I understand that the proposed solution would work for a "one-to-one" case where many files are landing in a specific dbfs path to ...

  • 1 kudos
2 More Replies
Cosmin
by • New Contributor II
  • 918 Views
  • 3 replies
  • 0 kudos

Fail to write large dataframe

Hi all, we have a issue while trying to write a quite large data frame, close to 35 million records. We try to write it as parquet and also table and none work. But writing a small chink (10k records) is working. Basically we have some text on which ...

Cosmin_2-1702640369404.png
Get Started Discussions
Reply
  • 918 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

That could work, but you will have to create a UDF.Check this SO topic for more info

  • 0 kudos
2 More Replies
vrajesh123
by • New Contributor II
  • 2289 Views
  • 2 replies
  • 0 kudos

Resolved! Webassessor Secure Browser will not Launch during exam.

Hello - I registered for the Databricks Data Engineering Associate Certification exam.  I hit an issue, their Secure browser would not launch, it just crashed - the only thing I could see in a flash is "bad request" and poof its gone.  Spend over 2 h...

Get Started Discussions
Certification Exam
Issues
Secure Browser
Reply
  • 2289 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @vrajesh123,    Thank you for posting your concern on Community! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 ho...

  • 0 kudos
1 More Replies
sudhanshu1
by • New Contributor III
  • 3442 Views
  • 3 replies
  • 2 kudos

Python file testing using pytest

Hi All,I have a requirement in my project, where we will be writing some python code inside databricks . Please note we will not be using pyspark . It will plain pythin with polars.I am looking into ho to create test files for main file. Below is sim...

Get Started Discussions
Reply
  • 3442 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @sudhanshu1 , The error you're encountering is due to the fact that pytest is looking for a fixture named 'name' which is not defined. Fixtures in pytest are functions that are run before each test function to which it is applied. Fixtures are us...

  • 2 kudos
2 More Replies
pablobd
by • Contributor
  • 1556 Views
  • 2 replies
  • 1 kudos

Resolved! Asset bundle build and deploy python wheel with versions

Hi all,I was able to deploy a wheel to the /Shared/ folder from a repository in Gitlab with asset bundles. The databricks.yml looks something like this.artifacts:  default:    type: whl    build: poetry build    path: .  targets:    workspace:      h...

Get Started Discussions
Reply
  • 1556 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
pablobd
Contributor
  • 1 kudos

Finally I decided to use AWS Code Artifact and mirror the PyPI, which I think it's a bit cleaner. But your solution looks good too. Thanks!

  • 1 kudos
1 More Replies
Phani1
by • Valued Contributor
  • 522 Views
  • 1 replies
  • 0 kudos

informatica jobs from data bricks

Hi TeamHow can we call informatica jobs from data bricks? could you please suggest on this.Regards,Phanindra

Get Started Discussions
informatica jobs
Reply
  • 522 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Phani1, Informatica and Databricks have a partnership that allows for seamless integration between the two platforms.   Here’s a high-level overview of how you can use Informatica with Databricks:   Data Ingestion: You can ingest data directly in...

  • 0 kudos
leelee3000
by • New Contributor III
  • 722 Views
  • 2 replies
  • 0 kudos

time travel with DLT

Needed some help with Time Travel with Delta Live tables   We were trying to figure out if we can go in and alter the history on this table, and what would happen to data that we mass upload?  By this we mean we have data from the past that we would ...

Get Started Discussions
Reply
  • 722 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
BigRoux
New Contributor III
  • 0 kudos

Delta Live Tables leverage Delta Lake, or Delta Tables.  Delta tables, through transactions (e.g. insert, update, delete, merges, optimization) create versions of said Delta Table.  Once a version is created it cannot be altered, it is immutable.  Yo...

  • 0 kudos
1 More Replies
Phani1
by • Valued Contributor
  • 741 Views
  • 2 replies
  • 0 kudos

Upgrade Spark version 3.2 to 3.4+

Hi Team,We would like to upgrade from Spark version 3.2 to 3.4+ (Databricks Runtime - 10.4 to 12.2/13.3)We would like to understand how complex upgradation is this and challenges which we face? what are the technical steps and precautions we need to ...

Get Started Discussions
spark
Reply
  • 741 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Phani1,   Check Databricks Runtime release notes for any specific changes, improvements, or deprecations that might impact your workloads. Our End-of-Year Community Survey is here! Please take a few moments to complete the survey. Your feedback m...

  • 0 kudos
1 More Replies
RamanP9404
by • New Contributor
  • 975 Views
  • 1 replies
  • 0 kudos

Spark Streaming Issues while performing left join

Hi team,I'm struck in a Spark Structured streaming use-case.Requirement: To read two streaming data frames, perform a left join on it and display the results. Issue: While performing a left join, the resultant data frame contains only rows where ther...

Get Started Discussions
Reply
  • 975 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @RamanP9404, In Spark Structured Streaming, watermarking is essential for handling late data and ensuring correctness in stream-stream joins. When performing a left outer join, both the left and right datasets must be watermarked with a time const...

  • 0 kudos
Phani1
by • Valued Contributor
  • 1203 Views
  • 1 replies
  • 0 kudos

Customer Managed Keys in Databricks (AWS)

Hi Databricks Team,Could you please provide me the detailed steps on how to be enabled customer managed keys in databricks (AWS) Account, if there is any video on it that would be great helpful.Regards,Phanindra

Get Started Discussions
Reply
  • 1203 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Phani1, Enabling customer-managed keys (CMKs) in Databricks on AWS involves securing and controlling access to encrypted data.    Here are the steps to set up CMKs:   Prerequisites: Ensure your Databricks workspace is on the E2 version of the Dat...

  • 0 kudos
dZegpi
by • New Contributor II
  • 649 Views
  • 3 replies
  • 0 kudos

Load GCP data to Databricks using R

I'm working with Databricks and Google Cloud in the same project. I want to load specific datasets stored in GCP into a R notebook in Databricks. I currently can see the datasets in BigQuery. The problem is that using the sparklyr package, I'm not ab...

Get Started Discussions
Reply
  • 649 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Our End-of-Year Community Survey is here! Please take a few moments to complete the survey. Your feedback matters!

  • 0 kudos
2 More Replies
vish93
by • New Contributor
  • 332 Views
  • 0 replies
  • 0 kudos

Best AI Art Generator

AI art generator uses artificial intelligence to create captivating artworks, redefining the boundaries of traditional creativity and enabling endless artistic possibilities.AI photo restoration is a groundbreaking technology that employs artificial ...

Get Started Discussions
Reply
  • 332 Views
  • 0 replies
  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors
View all