cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

ChristianRRL
by Valued Contributor III
  • 1033 Views
  • 0 replies
  • 0 kudos

Source to Bronze Organization + Partition

Hi there, I hope I have what is effectively a simple question. I'd like to ask for a bit on guidance if I am structuring my source-to-bronze auto loader data properly. Here's what I have currently:/adls_storage/<data_source_name>/<category>/autoloade...

  • 1033 Views
  • 0 replies
  • 0 kudos
pablobd
by Contributor II
  • 4472 Views
  • 2 replies
  • 0 kudos

Install python package from private repo [CodeArtifact]

As part of my MLOps stack, I have developed a few packages which are the published to a private AWS CodeArtifact repo. How can I connect the AWS CodeArtifact repo to databricks? I want to be able to add these packages to the requirements.txt of a mod...

  • 4472 Views
  • 2 replies
  • 0 kudos
Latest Reply
pablobd
Contributor II
  • 0 kudos

One way to do it is to run this line before installing the dependencies:pip config set site.index-url https://aws:$CODEARTIFACT_AUTH_TOKEN@my_domain-111122223333.d.codeartifact.region.amazonaws.com/pypi/my_repo/simple/But can we add this in MLFlow?

  • 0 kudos
1 More Replies
xfun
by New Contributor II
  • 2178 Views
  • 1 replies
  • 0 kudos

SQL Warehouse cluster is always running when configure metabase conneciton

I encountered an issue while using the Metabase JDBC driver to connect to Databricks SQL Warehouse: I noticed that the SQL Warehouse cluster is always running and never stops automatically. Every few seconds, a SELECT 1 query log appears, which I sus...

  • 2178 Views
  • 1 replies
  • 0 kudos
Latest Reply
xfun
New Contributor II
  • 0 kudos

Will try to remove "preferredTestQuery" or add "idleConnectionTestPeriod" to avoid keep send select 1 query. 

  • 0 kudos
OU_Professor
by New Contributor II
  • 2052 Views
  • 0 replies
  • 0 kudos

Is the Databricks Community Edition still available?

Hello,I am a professor of IT in the Price College of Business at The University of Oklahoma.  In the past, I have used the Databricks Community Edition to demonstrate the principles of building and maintaining a data warehouse.  This spring semester ...

  • 2052 Views
  • 0 replies
  • 0 kudos
pablobd
by Contributor II
  • 1622 Views
  • 2 replies
  • 1 kudos

utils.add_libraries_to_model creates a duplicated model

Hello,When I call this function,mlflow.models.utils.add_libraries_to_model(MODEL_URI)It register a new model into the Model Registry. Is it possible to do the same but without registering a new model?Thanks,

  • 1622 Views
  • 2 replies
  • 1 kudos
Latest Reply
pablobd
Contributor II
  • 1 kudos

I ended up publishing the library to AWS CodeArtifact repository. Now, how can I tell MLFlow to use AWS CodeArtifact private repository instead of PyPi?

  • 1 kudos
1 More Replies
ChristianRRL
by Valued Contributor III
  • 2739 Views
  • 2 replies
  • 0 kudos

Auto Loader Use Case Question - Centralized Dropzone to Bronze?

Good day,I am trying to use Auto Loader (potentially extending into DLT in the future) to easily pull data coming from an external system (currently located in a single location) and organize it and load it respectively. I am struggling quite a bit a...

  • 2739 Views
  • 2 replies
  • 0 kudos
Latest Reply
ChristianRRL
Valued Contributor III
  • 0 kudos

Quick follow-up on this @Retired_mod (or to anyone else in the Databricks multi-verse who is able to help clarify this case).I understand that the proposed solution would work for a "one-to-one" case where many files are landing in a specific dbfs pa...

  • 0 kudos
1 More Replies
Cosmin
by New Contributor II
  • 3025 Views
  • 3 replies
  • 0 kudos

Fail to write large dataframe

Hi all, we have a issue while trying to write a quite large data frame, close to 35 million records. We try to write it as parquet and also table and none work. But writing a small chink (10k records) is working. Basically we have some text on which ...

Cosmin_2-1702640369404.png
  • 3025 Views
  • 3 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

That could work, but you will have to create a UDF.Check this SO topic for more info

  • 0 kudos
2 More Replies
JordiDekker
by New Contributor III
  • 4560 Views
  • 2 replies
  • 0 kudos

ClassCastException when attempting to timetravel (databricks-connect)

Hi all,Using databricks-connect 11.3.19, I get an "java.lang.ClassCastException" when attempting to timetravel. The exact same statement works fine when executed in the databricks GUI directly. Any ideas on what's going on? Is this a known limitation...

  • 4560 Views
  • 2 replies
  • 0 kudos
Latest Reply
SusanaD
New Contributor II
  • 0 kudos

Did you find a solution?

  • 0 kudos
1 More Replies
kartik-chandra
by New Contributor III
  • 3502 Views
  • 2 replies
  • 0 kudos

Resolved! Spark read with format as "delta" isn't working with Java multithreading

0I have a Spark application (using Java library) which needs to replicate data from one blob storage to another. I have created a readStream() within it which is listening continuously to a Kafka topic for incoming events. The corresponding writeStre...

  • 3502 Views
  • 2 replies
  • 0 kudos
Latest Reply
kartik-chandra
New Contributor III
  • 0 kudos

The problem was indeed with the way ClassLoader was being set in the ForkJoinPool (common Pool used) thread. Spark in SparkClassUtils uses Thread.currentThread().getContextClassLoader which might behave differently in another thread.To solve it I cre...

  • 0 kudos
1 More Replies
vrajesh123
by New Contributor II
  • 6132 Views
  • 1 replies
  • 0 kudos

Webassessor Secure Browser will not Launch during exam.

Hello - I registered for the Databricks Data Engineering Associate Certification exam.  I hit an issue, their Secure browser would not launch, it just crashed - the only thing I could see in a flash is "bad request" and poof its gone.  Spend over 2 h...

Get Started Discussions
Certification Exam
Issues
Secure Browser
  • 6132 Views
  • 1 replies
  • 0 kudos
A1459
by New Contributor
  • 1321 Views
  • 0 replies
  • 0 kudos

Execute delete query from notebook on azure synapse

Hello Everyone, Is there a way we can execute the delete query from azure notebook on azure synapse database.I tried using read api method with option "query" but getting error like jdbc connector not able to handle code.Can any suggest how we can de...

  • 1321 Views
  • 0 replies
  • 0 kudos
sudhanshu1
by New Contributor III
  • 9610 Views
  • 2 replies
  • 2 kudos

Python file testing using pytest

Hi All,I have a requirement in my project, where we will be writing some python code inside databricks . Please note we will not be using pyspark . It will plain pythin with polars.I am looking into ho to create test files for main file. Below is sim...

  • 9610 Views
  • 2 replies
  • 2 kudos
Sujitha
by Databricks Employee
  • 11525 Views
  • 1 replies
  • 5 kudos

New how-to guide to data warehousing with the Data Intelligence Platform

Just launched: The Big Book of Data Warehousing and BI, a new hands-on guide focused on real-world use cases from governance, transformation, analytics and AI.As the demand for data becomes insatiable in every company, the data infrastructure has bec...

Screenshot 2023-12-05 at 11.04.08 AM.png
  • 11525 Views
  • 1 replies
  • 5 kudos
Latest Reply
Edward3
New Contributor II
  • 5 kudos

lol beans It used to take me a long time to regain my equilibrium, but recently I learned that a website really leads this layout when you may find delight after a stressful day here. Since then, I've been able to find my equilibrium much more quickl...

  • 5 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels