cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

pablobd
by Contributor II
  • 4324 Views
  • 2 replies
  • 0 kudos

Install python package from private repo [CodeArtifact]

As part of my MLOps stack, I have developed a few packages which are the published to a private AWS CodeArtifact repo. How can I connect the AWS CodeArtifact repo to databricks? I want to be able to add these packages to the requirements.txt of a mod...

  • 4324 Views
  • 2 replies
  • 0 kudos
Latest Reply
pablobd
Contributor II
  • 0 kudos

One way to do it is to run this line before installing the dependencies:pip config set site.index-url https://aws:$CODEARTIFACT_AUTH_TOKEN@my_domain-111122223333.d.codeartifact.region.amazonaws.com/pypi/my_repo/simple/But can we add this in MLFlow?

  • 0 kudos
1 More Replies
xfun
by New Contributor II
  • 2101 Views
  • 1 replies
  • 0 kudos

SQL Warehouse cluster is always running when configure metabase conneciton

I encountered an issue while using the Metabase JDBC driver to connect to Databricks SQL Warehouse: I noticed that the SQL Warehouse cluster is always running and never stops automatically. Every few seconds, a SELECT 1 query log appears, which I sus...

  • 2101 Views
  • 1 replies
  • 0 kudos
Latest Reply
xfun
New Contributor II
  • 0 kudos

Will try to remove "preferredTestQuery" or add "idleConnectionTestPeriod" to avoid keep send select 1 query. 

  • 0 kudos
OU_Professor
by New Contributor II
  • 1775 Views
  • 0 replies
  • 0 kudos

Is the Databricks Community Edition still available?

Hello,I am a professor of IT in the Price College of Business at The University of Oklahoma.  In the past, I have used the Databricks Community Edition to demonstrate the principles of building and maintaining a data warehouse.  This spring semester ...

  • 1775 Views
  • 0 replies
  • 0 kudos
pablobd
by Contributor II
  • 1552 Views
  • 2 replies
  • 1 kudos

utils.add_libraries_to_model creates a duplicated model

Hello,When I call this function,mlflow.models.utils.add_libraries_to_model(MODEL_URI)It register a new model into the Model Registry. Is it possible to do the same but without registering a new model?Thanks,

  • 1552 Views
  • 2 replies
  • 1 kudos
Latest Reply
pablobd
Contributor II
  • 1 kudos

I ended up publishing the library to AWS CodeArtifact repository. Now, how can I tell MLFlow to use AWS CodeArtifact private repository instead of PyPi?

  • 1 kudos
1 More Replies
ChristianRRL
by Valued Contributor III
  • 2641 Views
  • 2 replies
  • 0 kudos

Auto Loader Use Case Question - Centralized Dropzone to Bronze?

Good day,I am trying to use Auto Loader (potentially extending into DLT in the future) to easily pull data coming from an external system (currently located in a single location) and organize it and load it respectively. I am struggling quite a bit a...

  • 2641 Views
  • 2 replies
  • 0 kudos
Latest Reply
ChristianRRL
Valued Contributor III
  • 0 kudos

Quick follow-up on this @Retired_mod (or to anyone else in the Databricks multi-verse who is able to help clarify this case).I understand that the proposed solution would work for a "one-to-one" case where many files are landing in a specific dbfs pa...

  • 0 kudos
1 More Replies
Cosmin
by New Contributor II
  • 2913 Views
  • 3 replies
  • 0 kudos

Fail to write large dataframe

Hi all, we have a issue while trying to write a quite large data frame, close to 35 million records. We try to write it as parquet and also table and none work. But writing a small chink (10k records) is working. Basically we have some text on which ...

Cosmin_2-1702640369404.png
  • 2913 Views
  • 3 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

That could work, but you will have to create a UDF.Check this SO topic for more info

  • 0 kudos
2 More Replies
JordiDekker
by New Contributor III
  • 3541 Views
  • 2 replies
  • 0 kudos

ClassCastException when attempting to timetravel (databricks-connect)

Hi all,Using databricks-connect 11.3.19, I get an "java.lang.ClassCastException" when attempting to timetravel. The exact same statement works fine when executed in the databricks GUI directly. Any ideas on what's going on? Is this a known limitation...

  • 3541 Views
  • 2 replies
  • 0 kudos
Latest Reply
SusanaD
New Contributor II
  • 0 kudos

Did you find a solution?

  • 0 kudos
1 More Replies
kartik-chandra
by New Contributor III
  • 3371 Views
  • 2 replies
  • 0 kudos

Resolved! Spark read with format as "delta" isn't working with Java multithreading

0I have a Spark application (using Java library) which needs to replicate data from one blob storage to another. I have created a readStream() within it which is listening continuously to a Kafka topic for incoming events. The corresponding writeStre...

  • 3371 Views
  • 2 replies
  • 0 kudos
Latest Reply
kartik-chandra
New Contributor III
  • 0 kudos

The problem was indeed with the way ClassLoader was being set in the ForkJoinPool (common Pool used) thread. Spark in SparkClassUtils uses Thread.currentThread().getContextClassLoader which might behave differently in another thread.To solve it I cre...

  • 0 kudos
1 More Replies
vrajesh123
by New Contributor II
  • 5978 Views
  • 1 replies
  • 0 kudos

Webassessor Secure Browser will not Launch during exam.

Hello - I registered for the Databricks Data Engineering Associate Certification exam.  I hit an issue, their Secure browser would not launch, it just crashed - the only thing I could see in a flash is "bad request" and poof its gone.  Spend over 2 h...

Get Started Discussions
Certification Exam
Issues
Secure Browser
  • 5978 Views
  • 1 replies
  • 0 kudos
A1459
by New Contributor
  • 1271 Views
  • 0 replies
  • 0 kudos

Execute delete query from notebook on azure synapse

Hello Everyone, Is there a way we can execute the delete query from azure notebook on azure synapse database.I tried using read api method with option "query" but getting error like jdbc connector not able to handle code.Can any suggest how we can de...

  • 1271 Views
  • 0 replies
  • 0 kudos
sudhanshu1
by New Contributor III
  • 9499 Views
  • 2 replies
  • 2 kudos

Python file testing using pytest

Hi All,I have a requirement in my project, where we will be writing some python code inside databricks . Please note we will not be using pyspark . It will plain pythin with polars.I am looking into ho to create test files for main file. Below is sim...

  • 9499 Views
  • 2 replies
  • 2 kudos
Sujitha
by Databricks Employee
  • 11466 Views
  • 1 replies
  • 5 kudos

New how-to guide to data warehousing with the Data Intelligence Platform

Just launched: The Big Book of Data Warehousing and BI, a new hands-on guide focused on real-world use cases from governance, transformation, analytics and AI.As the demand for data becomes insatiable in every company, the data infrastructure has bec...

Screenshot 2023-12-05 at 11.04.08 AM.png
  • 11466 Views
  • 1 replies
  • 5 kudos
Latest Reply
Edward3
New Contributor II
  • 5 kudos

lol beans It used to take me a long time to regain my equilibrium, but recently I learned that a website really leads this layout when you may find delight after a stressful day here. Since then, I've been able to find my equilibrium much more quickl...

  • 5 kudos
pablobd
by Contributor II
  • 3199 Views
  • 1 replies
  • 0 kudos

Asset bundle build and deploy python wheel with versions

Hi all,I was able to deploy a wheel to the /Shared/ folder from a repository in Gitlab with asset bundles. The databricks.yml looks something like this.artifacts:  default:    type: whl    build: poetry build    path: .  targets:    workspace:      h...

  • 3199 Views
  • 1 replies
  • 0 kudos
Latest Reply
pablobd
Contributor II
  • 0 kudos

Finally I decided to use AWS Code Artifact and mirror the PyPI, which I think it's a bit cleaner. But your solution looks good too. Thanks!

  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels