- 9962 Views
- 2 replies
- 2 kudos
Resolved! requirements.txt with cluster libraries
Cluster libraries are supported from version 15.0 - Databricks Runtime 15.0 | Databricks on AWS.How can I specify requirements.txt file path in the libraries in a job cluster in my workflow? Can I use relative path? Is it relative from the root of th...
- 9962 Views
- 2 replies
- 2 kudos
- 2 kudos
how to install requirement.txt using github action.- name: Install workspace requirements.txt on clusterenv:CLUSTER_ID: ${{ secrets.DATABRICKS_CLUSTER_ID }}run: |databricks libraries install \--cluster-id "$CLUSTER_ID" \--whl "dbfs:/FileStore/enginee...
- 2 kudos
- 3585 Views
- 5 replies
- 0 kudos
Debugging notebook access to external REST API
I'm using a Python Notebook with a REST API to access a system outside Databricks, in this case it's to call a SAS program. Identical python code works fine if I call it from jupyter on my laptop, but fails with a timeout when I run it from my Databr...
- 3585 Views
- 5 replies
- 0 kudos
- 0 kudos
What happens if you run command in notebook:nc -vz hostname 443If it fails to connect this will mean that the firewall or security groups associated with the VPC or VNet are not allowing this connection, you will need to check with your networking te...
- 0 kudos
- 2145 Views
- 1 replies
- 0 kudos
Trusted assets vs query examples
¡Hi community! In recent days I explored trusted assets in my genie space and this working very well! but I feel a little confused :sIn my genie space I have many queries examples when I create a new function with the same query example for verify th...
- 2145 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Dulce42! It depends on your use case. If your function covers the scenario well, you don’t need a separate query example. Having both for the same purpose can create redundancy and make things more complex. Choose the option that best fits you...
- 0 kudos
- 2442 Views
- 2 replies
- 0 kudos
Resolved! Need help to add personal email to databricks partner account
I have been actively using the Databricks Partner Academy for the past three years through my current organization. As I am planning to transition to a new company, I would like to ensure continued access to my training records and certifications.Cur...
- 2442 Views
- 2 replies
- 0 kudos
- 1876 Views
- 1 replies
- 0 kudos
Python versions - Notebooks and DBR
Hi,I have a problem with conflicting python versions in a notebook running with the Databricks 14 day free trial. One example:spark.conf.get("spark.databricks.clusterUsageTags.clusterName") # Returns: "Python versions in the Spark Connect client and...
- 1876 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Terje, were you able to fix it? From what I know, during the free trial period we’re limited to the default setup, so version mismatches can’t be resolved unless we upgrade to a paid workspace.
- 0 kudos
- 1076 Views
- 1 replies
- 1 kudos
Completed Machine learning course
I have completed my course for Machine learning as part of Learning festival.
- 1076 Views
- 1 replies
- 1 kudos
- 1 kudos
Great to hear, @faridaeasmin‌ ‌.
- 1 kudos
- 2448 Views
- 2 replies
- 0 kudos
Python coding in notebook with a (long) token
I have written a python program (called by a trigger) that uses a token issued by a third party app (it's circa 400 bytes long including '.' and '-'). When I copy/paste this token into a Databricks notebook - curious formatting takes place and a coup...
- 2448 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey Paul, You can use databricks secrets for preserving the integrity of the token.Here's the databricks doc for refernece : https://docs.databricks.com/aws/en/security/secrets
- 0 kudos
- 5626 Views
- 3 replies
- 0 kudos
Save output of show table extended to table?
I want to save the output of show table extended in catalogName like 'mysearchtext*';to a table.How do I do that?
- 5626 Views
- 3 replies
- 0 kudos
- 0 kudos
Use DESCRIBE EXTENDED customer AS JSON this returns as a json data . This you can load Applicable to databricks 16.2 and abovehttps://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-aux-describe-table
- 0 kudos
- 6827 Views
- 2 replies
- 1 kudos
Missing Genie - Upload File Feature in Preview Section
Despite having admin privileges for both the workspace and Genie Workspace, we are unable to see the "Genie - Upload File" feature under the Preview section, even though the documentation indicates it should be available.We also attempted switching r...
- 6827 Views
- 2 replies
- 1 kudos
- 1 kudos
For more information around upload a file option please refer https://docs.databricks.com/aws/en/genie/file-uploadit supports csv and excel datasets as of now with condition that files must be smaller than 200 MB and contain fewer than 100 columns du...
- 1 kudos
- 4203 Views
- 4 replies
- 4 kudos
Resolved! using Azure Databricks vs using Databricks directly
Hi friends,A quick question regarding how data, workspace controls works while using "Azure Databricks". I am planning to use Azure Databricks that comes as part of my employer's Azure Subscriptions. I work for a Public sector organization, which is ...
- 4203 Views
- 4 replies
- 4 kudos
- 1051 Views
- 1 replies
- 0 kudos
Right course for ML engineer
Hi I would like to learn databricks so that I could look for job opportunities as a ML engineer. I have background with python programming, computer vision (OpenCV) .not having much of experience with azure , aws so on.which course here is good with ...
- 1051 Views
- 1 replies
- 0 kudos
- 0 kudos
Given your background in Python programming and computer vision but limited experience with cloud platforms, the best pathway to enter the job market as MLE using Databricks is to pursue the Databricks Certified Machine Learning Associate certificati...
- 0 kudos
- 4967 Views
- 1 replies
- 1 kudos
Resolved! Why does .collect() cause a shuffle while .show() does not?
I’m learning Spark using the book Spark: The Definitive Guide and came across some behavior I’m trying to understand.I am reading a csv_file which has 3 columns: DEST_COUNTRY_NAME, ORIGIN_COUNTRY_NAME, count. The dataset has a total of 256 rows.Here’...
- 4967 Views
- 1 replies
- 1 kudos
- 1 kudos
Q1: collect() moves all data to the driver, hence a shufle. show() just shows x records from the df, from a partition (or more partitions if x > partition size). No shuffling needed.For display purposes the results are of course gathered on the driv...
- 1 kudos
- 2486 Views
- 2 replies
- 2 kudos
Lazy evaluation in serverless vs all purpose compute ?
As you can see right now I am connected to serverless compute and when I give wrong path, spark does lazy evaluation and gives error on display. However, when I switch from serverless to my all purpose cluster I get the error when I create the df its...
- 2486 Views
- 2 replies
- 2 kudos
- 2 kudos
Based on the scenario, what https://community.databricks.com/t5/user/viewprofilepage/user-id/156441 saying is correct though the eager evaluation property is false in both cases and for All-Purpose clusters, Spark is checking the path immediately whe...
- 2 kudos
- 1620 Views
- 1 replies
- 0 kudos
Unable to access external table created by DLT
I originally set the Storage location in my DLT as abfss://{container}@{storageaccount}.dfs.core.windows.net/...But when running the DLT I got the following error:So I decided to leave the above Storage location blank and define the path parameter in...
- 1620 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Tommy , Thanks for your question. I would encourage you to verify once using a Pro SQL Warehouse temporarily instead of a Serverless SQL Warehouse given the compute differences between the two - Pro compute resides in your data plane, Serverless ...
- 0 kudos
- 4341 Views
- 2 replies
- 2 kudos
OCRmyPDF in Databricks
Hello,Do any of you have experience with using OCRmyPDF in Databricks? I have tried to install it in various was with different versions, but my notebook keep crashing with the error:The Python process exited with exit code 139 (SIGSEGV: Segmentation...
- 4341 Views
- 2 replies
- 2 kudos
- 2 kudos
Refer to this link too https://community.databricks.com/t5/data-engineering/pdf-parsing-in-notebook/td-p/14636
- 2 kudos
