cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

ChristianRRL
by Valued Contributor III
  • 9514 Views
  • 5 replies
  • 2 kudos

DLT Compute Resources - What Compute Is It???

Hi there, I'm wondering if someone can help me understand what compute resources DLT uses? It's not clear to me at all if it uses the last compute cluster I had been working on, or something else entirely.Can someone please help clarify this? 

ChristianRRL_0-1703025826377.png
  • 9514 Views
  • 5 replies
  • 2 kudos
Latest Reply
quakenbush
Contributor
  • 2 kudos

Well, one thing they emphasize in the 'Adavanced Data Engineer' Training is that job-clusters will terminate within 5 minutes after a job is completed. So this could be in support of your theory to lower costs. I think job-cluster are actually design...

  • 2 kudos
4 More Replies
chari
by Contributor
  • 2087 Views
  • 1 replies
  • 0 kudos

python library in databricks

Hello community members,I am seeking to understand where databricks keeps all the python libraries ? For a start, I tried two lines below:import sys sys.path()This list all the paths but I cant look inside them. How is DBFS different from these paths...

  • 2087 Views
  • 1 replies
  • 0 kudos
Latest Reply
Wojciech_BUK
Valued Contributor III
  • 0 kudos

Hello,all your libraries are installed on Databricks Cluster Driver node on OS Disk.DBFS is like mounted Cloude Storage account.You have veriouse ways of working with libraries but databricks only load some of libraries that comes with Cluster image....

  • 0 kudos
SamGreene
by Contributor II
  • 3001 Views
  • 1 replies
  • 0 kudos

Power BI keeps SQL Warehouse Running

Hi,I have a SQL Warehouse, serverless mode, set to shut down after 5 minutes.  Using the databricks web IDE, this works as expected.  However, if I connect Power BI, import data to PBI and then leave the application open, the SQL Warehouse does not s...

  • 3001 Views
  • 1 replies
  • 0 kudos
Latest Reply
SamGreene
Contributor II
  • 0 kudos

Repeating this test today, the SQL Warehouse shut down properly.  Thanks for your helpful reply. 

  • 0 kudos
Lotfi
by New Contributor II
  • 2632 Views
  • 2 replies
  • 0 kudos

Seeking Assistance with Dynamic %run Command Path

Hello Databricks Community Team,I trust this message finds you well. I am currently facing an issue while attempting to utilize a dynamic path with the %run command to execute a notebook called from another folder. I have tested the following approac...

Get Started Discussions
Notebook
path
run
  • 2632 Views
  • 2 replies
  • 0 kudos
Latest Reply
chari
Contributor
  • 0 kudos

Hi, If your config file is in the databricks file system then you should add dbfs:/Ex: f"dbfs:/Users/.../blob_conf/{conf_file}"

  • 0 kudos
1 More Replies
Ramakrishnan83
by New Contributor III
  • 809 Views
  • 0 replies
  • 0 kudos

How to Migrate specific notebooks from one Azure Repo to another Azure Repo

Team,I need to migrate only specific notebooks which has changes committed to be pulled from one repo to another repoEnvironment/Repo Setup:Master  -> Dev -> Feature Branch -> Developer commits the code in Feature Branch -> Dev has the changes from D...

  • 809 Views
  • 0 replies
  • 0 kudos
Oleksandr
by New Contributor II
  • 1043 Views
  • 1 replies
  • 0 kudos

Databricks JDBC Driver 2.6.36 includes dependencies in pom.properties with vulnerabilities

Starting from Databricks JDBC Driver 2.6.36 we've got Trivy security report with vulnerabilities from pom.properties.2.6.36 adds org.apache.commons.commons-compress:1.20 and ch.qos.logback.logback-classic:1.2.3.2.6.34 doesn't include such dependencie...

  • 1043 Views
  • 1 replies
  • 0 kudos
Latest Reply
Oleksandr
New Contributor II
  • 0 kudos

I didn't find where to open an issue (GitHub or Jira). Please, let me know if I need to report it somewhere else.

  • 0 kudos
_YSF
by New Contributor II
  • 7023 Views
  • 2 replies
  • 2 kudos

UC Volumes - Cannot access the UC Volume path from this location. Path was

Hi, I'm trying out the new Volumes preview.I'm using external locations for everything so far. I have my storage credential, and external locations created and tested. I created a catalog, schema and in that schema a volume. In the new data browser o...

  • 7023 Views
  • 2 replies
  • 2 kudos
Latest Reply
franciscomorra
New Contributor II
  • 2 kudos

Hope this helps, but this issue could be caused by the Cluster being in no-isolation shared and not in single-user or shared, both compatible with Unity Catalog

  • 2 kudos
1 More Replies
Junaid_Ali
by New Contributor II
  • 1180 Views
  • 0 replies
  • 0 kudos

Creating external location is Failing because of cross plane request

While creating Unity Catalog external location from Data Bricks UI or from a notebook using "CREATE EXTERNAL LOCATION location_name .." a connection is being made and rejected from control plane to the S3 data bucket in a PrivateLink enabled environm...

Get Started Discussions
Unity Catalog
VPC
  • 1180 Views
  • 0 replies
  • 0 kudos
ChristianRRL
by Valued Contributor III
  • 1024 Views
  • 0 replies
  • 0 kudos

Source to Bronze Organization + Partition

Hi there, I hope I have what is effectively a simple question. I'd like to ask for a bit on guidance if I am structuring my source-to-bronze auto loader data properly. Here's what I have currently:/adls_storage/<data_source_name>/<category>/autoloade...

  • 1024 Views
  • 0 replies
  • 0 kudos
pablobd
by Contributor II
  • 4445 Views
  • 2 replies
  • 0 kudos

Install python package from private repo [CodeArtifact]

As part of my MLOps stack, I have developed a few packages which are the published to a private AWS CodeArtifact repo. How can I connect the AWS CodeArtifact repo to databricks? I want to be able to add these packages to the requirements.txt of a mod...

  • 4445 Views
  • 2 replies
  • 0 kudos
Latest Reply
pablobd
Contributor II
  • 0 kudos

One way to do it is to run this line before installing the dependencies:pip config set site.index-url https://aws:$CODEARTIFACT_AUTH_TOKEN@my_domain-111122223333.d.codeartifact.region.amazonaws.com/pypi/my_repo/simple/But can we add this in MLFlow?

  • 0 kudos
1 More Replies
xfun
by New Contributor II
  • 2165 Views
  • 1 replies
  • 0 kudos

SQL Warehouse cluster is always running when configure metabase conneciton

I encountered an issue while using the Metabase JDBC driver to connect to Databricks SQL Warehouse: I noticed that the SQL Warehouse cluster is always running and never stops automatically. Every few seconds, a SELECT 1 query log appears, which I sus...

  • 2165 Views
  • 1 replies
  • 0 kudos
Latest Reply
xfun
New Contributor II
  • 0 kudos

Will try to remove "preferredTestQuery" or add "idleConnectionTestPeriod" to avoid keep send select 1 query. 

  • 0 kudos
OU_Professor
by New Contributor II
  • 1954 Views
  • 0 replies
  • 0 kudos

Is the Databricks Community Edition still available?

Hello,I am a professor of IT in the Price College of Business at The University of Oklahoma.  In the past, I have used the Databricks Community Edition to demonstrate the principles of building and maintaining a data warehouse.  This spring semester ...

  • 1954 Views
  • 0 replies
  • 0 kudos
pablobd
by Contributor II
  • 1613 Views
  • 2 replies
  • 1 kudos

utils.add_libraries_to_model creates a duplicated model

Hello,When I call this function,mlflow.models.utils.add_libraries_to_model(MODEL_URI)It register a new model into the Model Registry. Is it possible to do the same but without registering a new model?Thanks,

  • 1613 Views
  • 2 replies
  • 1 kudos
Latest Reply
pablobd
Contributor II
  • 1 kudos

I ended up publishing the library to AWS CodeArtifact repository. Now, how can I tell MLFlow to use AWS CodeArtifact private repository instead of PyPi?

  • 1 kudos
1 More Replies
ChristianRRL
by Valued Contributor III
  • 2726 Views
  • 2 replies
  • 0 kudos

Auto Loader Use Case Question - Centralized Dropzone to Bronze?

Good day,I am trying to use Auto Loader (potentially extending into DLT in the future) to easily pull data coming from an external system (currently located in a single location) and organize it and load it respectively. I am struggling quite a bit a...

  • 2726 Views
  • 2 replies
  • 0 kudos
Latest Reply
ChristianRRL
Valued Contributor III
  • 0 kudos

Quick follow-up on this @Retired_mod (or to anyone else in the Databricks multi-verse who is able to help clarify this case).I understand that the proposed solution would work for a "one-to-one" case where many files are landing in a specific dbfs pa...

  • 0 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels